[BUG] v3003 a salt-master using python 3.7+ using salt-ssh cannot work with a minion using python3.5 or 3.6

[bryceml]

Setup
install salt-ssh on an ubuntu 20.04 box and use it to control an ubuntu 16.04 box (probably happens with centos 7, 8, debian 9, and ubuntu 18.04 as well)

Steps to Reproduce the behavior

salt-ssh \* test.ping
minion-id:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.root_d24770_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.root_d24770_salt/pyall/salt/scripts.py", line 437, in salt_call
            import salt.cli.call
          File "/var/tmp/.root_d24770_salt/pyall/salt/cli/call.py", line 6, in <module>
            import salt.cli.caller
          File "/var/tmp/.root_d24770_salt/pyall/salt/cli/caller.py", line 14, in <module>
            import salt.loader
          File "/var/tmp/.root_d24770_salt/pyall/salt/loader.py", line 7, in <module>
            import contextvars
          File "/var/tmp/.root_d24770_salt/py3/contextvars.py", line 1, in <module>
            from _contextvars import Context, ContextVar, Token, copy_context
        ImportError: No module named '_contextvars'
    stdout:

I also tried doing pip3 install contextvars immutables --prefix /usr and pip3 install contextvars immutables on the master and it didn't fix the issue.

Expected behavior

minion-id:
    True

Versions Report

Salt Version:
          Salt: 3003
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.5 (default, Jan 27 2021, 15:41:15)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-70-generic
        system: Linux
       version: Ubuntu 20.04 focal
 

[sagetherage]

@frogunder @Ch3LL if we don't have a test for this in the packaging tests we should consider adding something see internal slack conversation starting here: https://saltstack.slack.com/archives/C04HEQ97D/p1617221478334300 in releasediscussions

[dwoz]

@frogunder @Ch3LL if we don't have a test for this in the packaging tests we should consider adding something see internal slack conversation starting here: https://saltstack.slack.com/archives/C04HEQ97D/p1617221478334300 in releasediscussions

Ideally we'd be able to add a test for this in the regular test suite too. With the docker stuff in pytest we should probably be able to come up with something.

[bryceml]

probably a different issue, but I also see

[ERROR   ] Pillar override was not passed as a dictionary
[ERROR   ] Pillar override was not passed as a dictionary
[ERROR   ] Pillar override was not passed as a dictionary
[ERROR   ] Pillar override was not passed as a dictionary
[ERROR   ] Pillar override was not passed as a dictionary
[ERROR   ] Pillar override was not passed as a dictionary
[ERROR   ] Pillar override was not passed as a dictionary
[ERROR   ] Pillar override was not passed as a dictionary
[ERROR   ] Pillar override was not passed as a dictionary

as well with salt-ssh where I didn't before when running salt-ssh \* state.single pkg.uptodate uptodate refresh=True dist_upgrade=True

debug output

[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] LazyLoaded state.single
[DEBUG   ] Gathering pillar data for state run
[ERROR   ] Pillar override was not passed as a dictionary
[DEBUG   ] Determining pillar cache
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Finished gathering pillar data for state run
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded pkg.install
[DEBUG   ] LazyLoaded pkg.uptodate
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points

[danmac-uk]

Can confirm with Debian 10 master, salt-ssh now fails with the contextvars error on Debian 9 target, as of the 3003 update. Debian 10 targets unaffected.

[djneades]

This bug also occurs against a Ubuntu 18.04 target.

[danmac-uk]

For those on Debian 10 who want things working, I recommend editing your apt sources.list file that references the saltstack repo (mine is /etc/apt/sources.list.d/saltstack.list) replace "latest" with "3002" at the end of the URL, then run the following:

apt-get update && apt-get install salt-master=3002.6+ds-1 salt-common=3002.6+ds-1 salt-ssh=3002.6+ds-1

This should stop any future upgrades >3002. If you feel lucky again, just revert the changes to the sources list file and upgrade as normal. Should work on Ubuntu also, but I didn't test that. Super paranoid should consider "apt-mark hold".

[auphofBSF]

Experiencing the same issue target minion (raspbian - stretch, python 3.5.3) salt-minion version 3003 installed
On target minion salt-call --local test.version responds correctly

on master using salt-ssh (3003) running in (alpine linux 3.13 python 3.7.10)
salt-ssh -i <DeviceID> -r 'ls -la' responds correctly
salt-ssh -i <DeviceID> test.varsion fails with ImportError: No module named '_contextvars'

[sagetherage]

Experiencing the same issue target minion (raspbian - stretch, python 3.5.3) salt-minion version 3003 installed
On target minion salt-call --local test.version responds correctly

on master using salt-ssh (3003) running in (alpine linux 3.13 python 3.7.10)
salt-ssh -i <DeviceID> -r 'ls -la' responds correctly
salt-ssh -i <DeviceID> test.varsion fails with ImportError: No module named '_contextvars'

@auphofBSF looking to correct in the 3003.1 minor release later in April

[xuhcc]

I installed contextvars package on a target but it didn't help.
salt uses contextvars module at /var/tmp/.***_salt/py3/contextvars.py instead of site-packages version.

[bryceml]

it has to be installed on the master, not the target. Also if you've already ran salt-ssh before installing contextvars, you may have to use the -w -t flags in order to re-generate the thin tarball.

@dwoz any other insight here?

also, please everyone report the os and install method of the box you are running salt-ssh from and whether the contextvars package is installed on that box.

[djneades]

@bryceml Thank you for the suggestions, but I’d already taken care of both of those. My salt-ssh is installed via python3 -m pip install (equivalent to pip3 install).

For the avoidance of doubt, to show installation, my master contextvars version, and everything else relevant …

On the master:

$ python3 -m pip install --user --upgrade salt==3003.1 mako
Requirement already satisfied: salt==3003.1 in /home/djn/.local/lib/python3.8/site-packages (3003.1)
Requirement already satisfied: mako in /home/djn/.local/lib/python3.8/site-packages (1.1.4)
Requirement already satisfied: contextvars in /home/djn/.local/lib/python3.8/site-packages (from salt==3003.1) (2.4)
Requirement already satisfied: pycryptodomex>=3.9.8 in /home/djn/.local/lib/python3.8/site-packages (from salt==3003.1) (3.9.9)
Requirement already satisfied: MarkupSafe in /usr/lib/python3/dist-packages (from salt==3003.1) (1.1.0)
Requirement already satisfied: msgpack!=0.5.5,>=0.5 in /home/djn/.local/lib/python3.8/site-packages (from salt==3003.1) (1.0.0)
Requirement already satisfied: pyzmq>=17.0.0 in /home/djn/.local/lib/python3.8/site-packages (from salt==3003.1) (20.0.0)
Requirement already satisfied: Jinja2 in /usr/lib/python3/dist-packages (from salt==3003.1) (2.10.1)
Requirement already satisfied: requests>=1.0.0 in /home/djn/.local/lib/python3.8/site-packages (from salt==3003.1) (2.25.1)
Requirement already satisfied: PyYAML in /usr/lib/python3/dist-packages (from salt==3003.1) (5.3.1)
Requirement already satisfied: distro>=1.0.1 in /home/djn/.local/lib/python3.8/site-packages (from salt==3003.1) (1.5.0)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests>=1.0.0->salt==3003.1) (2.8)
Requirement already satisfied: chardet<5,>=3.0.2 in /usr/lib/python3/dist-packages (from requests>=1.0.0->salt==3003.1) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=1.0.0->salt==3003.1) (2019.11.28)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests>=1.0.0->salt==3003.1) (1.25.8)
Requirement already satisfied: immutables>=0.9 in /home/djn/.local/lib/python3.8/site-packages (from contextvars->salt==3003.1) (0.15)

$ salt-ssh --versions
Salt Version:
          Salt: 3003.1

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.9.9
        pygit2: Not Installed
        Python: 3.8.5 (default, May 27 2021, 13:30:53)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.3

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.10.16.3-microsoft-standard-WSL2
        system: Linux
       version: Ubuntu 20.04 focal

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"

$ pip3 list | grep contextvars
contextvars            2.4

$ salt-ssh -w -t redacted.host.net test.ping
redacted.host.net:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.djn_165e34_salt/pyall/salt/loader_context.py", line 10, in <module>
            import _contextvars as contextvars
        ModuleNotFoundError: No module named '_contextvars'

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/var/tmp/.djn_165e34_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.djn_165e34_salt/pyall/salt/scripts.py", line 437, in salt_call
            import salt.cli.call
          File "/var/tmp/.djn_165e34_salt/pyall/salt/cli/call.py", line 6, in <module>
            import salt.cli.caller
          File "/var/tmp/.djn_165e34_salt/pyall/salt/cli/caller.py", line 14, in <module>
            import salt.loader
          File "/var/tmp/.djn_165e34_salt/pyall/salt/loader.py", line 26, in <module>
            import salt.config
          File "/var/tmp/.djn_165e34_salt/pyall/salt/config/__init__.py", line 103, in <module>
            _DFLT_IPC_WBUFFER = _gather_buffer_space() * 0.5
          File "/var/tmp/.djn_165e34_salt/pyall/salt/config/__init__.py", line 91, in _gather_buffer_space
            import salt.grains.core
          File "/var/tmp/.djn_165e34_salt/pyall/salt/grains/core.py", line 32, in <module>
            import salt.modules.cmdmod
          File "/var/tmp/.djn_165e34_salt/pyall/salt/modules/cmdmod.py", line 31, in <module>
            import salt.utils.templates
          File "/var/tmp/.djn_165e34_salt/pyall/salt/utils/templates.py", line 20, in <module>
            import salt.utils.jinja
          File "/var/tmp/.djn_165e34_salt/pyall/salt/utils/jinja.py", line 21, in <module>
            import salt.fileclient
          File "/var/tmp/.djn_165e34_salt/pyall/salt/fileclient.py", line 15, in <module>
            import salt.client
          File "/var/tmp/.djn_165e34_salt/pyall/salt/client/__init__.py", line 28, in <module>
            import salt.cache
          File "/var/tmp/.djn_165e34_salt/pyall/salt/cache/__init__.py", line 19, in <module>
            from salt.payload import Serial
          File "/var/tmp/.djn_165e34_salt/pyall/salt/payload.py", line 15, in <module>
            import salt.loader_context
          File "/var/tmp/.djn_165e34_salt/pyall/salt/loader_context.py", line 13, in <module>
            import contextvars
          File "/var/tmp/.djn_165e34_salt/py3/contextvars.py", line 1, in <module>
            from _contextvars import Context, ContextVar, Token, copy_context
        ModuleNotFoundError: No module named '_contextvars'
    stdout:

On the target:

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

# python3 --version
Python 3.6.9

# pip3 list | grep contextvars
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
contextvars (2.4)

# salt --versions

Command 'salt' not found, but can be installed with:

apt install salt-master

(I had manually installed contextvars on the target just in case it helped; it didn’t.)

[xuhcc]

@bryceml Installing contextvars on master indeed solved the issue for me. Thanks

I'm using arch linux and installed salt-ssh from the official repo (btw here's a bug report for the arch package https://bugs.archlinux.org/task/71344).

[djneades]

Rather than continue to fight this problem, I have updated my Linux targets to Ubuntu 20.04 LTS so that it will no longer occur. Thus, if I am the only person who was still seeing the issue, it can be safely ignored.

[aliasdevelopment]

Had the same issue running salt-ssh 3003.2 from Fedora 34 ( Python 3.9.6 ) to Centos 8 ( Python 3.6.8 ). Pip3 contextvars on master did not do anything.

I solved this temporarily by yum install salt-ssh-3002.6, until this is solved and released

[sp1ff]

Can confirm I'm still seeing this (Pop! OS, Python 3.9.5). pip install salt-ssh=3002.6 solved it.

[blackzombie87]

The problem still exist with 3004!
I've found a workaround, but maybe it's not the right way to "solve" this:

First, i needed to include typing_extensions into the thin package:

~# diff thin.py /usr/local/lib/python3.9/dist-packages/salt/utils/thin.py
30a31
> import typing_extensions
427a429
>         typing_extensions,

And second, the harder part if running on Python 3.5 (yes, ugly...) is a bug in immutables which can be solved with this:

~# diff _protocols.py /usr/local/lib/python3.9/dist-packages/immutables/_protocols.py
6c6
< from typing import NoReturn
---
> #from typing import NoReturn
15a16
>     from typing import NoReturn
18a20
>     from typing_extensions import NoReturn

It will move the import of NoReturn into the section which decides which version of Python is running, as NoReturn is not included in the typing extension of Python 3.5

Sorry for directly pasting diff's here, but i never did a merge before so maybe someone here is able to do that :)

[drigolin]

Yes is present also in 3004

[lastmikoi]

The problem still exist with 3004! I've found a workaround, but maybe it's not the right way to "solve" this:

First, i needed to include typing_extensions into the thin package:

~# diff thin.py /usr/local/lib/python3.9/dist-packages/salt/utils/thin.py
30a31
> import typing_extensions
427a429
>         typing_extensions,

Thank you VERY much for that workaround, I was able to workaround my issue and start working on my 10-month froze deployment (rockylinux is only supported on >3004 so rolling back to 3002 was impossible), after I remembered to clean my .cache folder.

I can confirm that, without this workaround, the issue is still here, even with a python 3.6 archlinux virtualenv trying to salt-ssh to a python 3.6 rockylinux 8 target.

[OrangeDog]

Reopening due to multiple reports of not being fixed.

[lastmikoi]

Thanks for the re-opening @OrangeDog !

I've gone ahead and troubleshooted the issue in detail, I've worked on a fix (#62234) which seems to be functional.

As an excerpt of my troubleshooting, from the PR:

[...] the contextvars backport makes uses of the immutables module.

This module went through a typing refactor [...] on August 2021.

However, that typing refactor added a new dependency, typing_extensions, but only on pre-3.8 deployments

[tao-devops]

Setup install salt-ssh on an ubuntu 20.04 box and use it to control an ubuntu 16.04 box (probably happens with centos 7, 8, debian 9, and ubuntu 18.04 as well)

Steps to Reproduce the behavior

salt-ssh \* test.ping
minion-id:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.root_d24770_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.root_d24770_salt/pyall/salt/scripts.py", line 437, in salt_call
            import salt.cli.call
          File "/var/tmp/.root_d24770_salt/pyall/salt/cli/call.py", line 6, in <module>
            import salt.cli.caller
          File "/var/tmp/.root_d24770_salt/pyall/salt/cli/caller.py", line 14, in <module>
            import salt.loader
          File "/var/tmp/.root_d24770_salt/pyall/salt/loader.py", line 7, in <module>
            import contextvars
          File "/var/tmp/.root_d24770_salt/py3/contextvars.py", line 1, in <module>
            from _contextvars import Context, ContextVar, Token, copy_context
        ImportError: No module named '_contextvars'
    stdout:

I also tried doing pip3 install contextvars immutables --prefix /usr and pip3 install contextvars immutables on the master and it didn't fix the issue.

Expected behavior

minion-id:
    True

Versions Report

Salt Version:
          Salt: 3003
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.5 (default, Jan 27 2021, 15:41:15)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-70-generic
        system: Linux
       version: Ubuntu 20.04 focal
 

as

[fs30000]

I had the the "ModuleNotFoundError: No module named 'contextvars'" error when connecting to a centos 7 machine using salt-ssh. Of course doing pip3 install contextvars on the target fixed it.

[Reonaydo]

Why Salt puts contextvars.py from Python-3.7+ into archive? If on Python-3.7+ on target system it already has his own contextvars.py. But if target system has lesser version of Python than 3.7, this contextvars.py breaks it.
loader.py has support for 3.6 and 3.7+ versions in next code

try:
    # Try the stdlib C extension first
    import _contextvars as contextvars
except ImportError:
    # Py<3.7
    import contextvars

So I think that salt mustn't put contextvars.py in thin.tgz
I think there should always be contextvars backport from PyPi instead(but I'm not sure how should it be put inside)
Python-3.7+ on target system will skip it in loader but legacy version will use this backported contextvars

I made this fast patch and is works for me for rhel7-based and rhel8-based systems if python3-contextvars is installed on those systems.

--- /usr/lib64/python3.10/site-packages/salt/utils/thin.py.orig      2023-01-30 12:16:01.000000000 +0300
+++ /usr/lib64/python3.10/site-packages/salt/utils/thin.py        2023-03-09 20:18:51.527031976 +0300
@@ -427,12 +427,14 @@
         backports_abc,
     ]
     modules = find_site_modules("contextvars")
+    contextvars = None
     if modules:
         contextvars = modules[0]
-    else:
+    elif py_contextvars.__file__.split('/')[-1] != 'contextvars.py':
         contextvars = py_contextvars
-    log.debug("Using contextvars %r", contextvars)
-    mods.append(contextvars)
+    if contextvars:
+        log.debug("Using contextvars %r", contextvars)
+        mods.append(contextvars)
     if has_immutables:
         mods.append(immutables)
     for mod in mods:

[Reonaydo]

Oh I got it. contextvars exists in dependencies but was removed but my distro maintainers.

[dmurphy18]

Closing this, since it appears to be fixed, please reopen if other points to raise