Skip to content

[BUG] salt-ssh fails due to attempting to pickle _thread._local object #61435

Closed

Description

@digitalresistor

Description

% env HOMEBREW_PREFIX=/opt/homebrew salt-ssh 'strida-prod' test.ping
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
TypeError: cannot pickle '_thread._local' object
Traceback (most recent call last):
  File "/opt/homebrew/bin/salt-ssh", line 8, in <module>
    sys.exit(salt_ssh())
  File "/opt/homebrew/Cellar/salt/3004/libexec/lib/python3.10/site-packages/salt/scripts.py", line 459, in salt_ssh
    client.run()
  File "/opt/homebrew/Cellar/salt/3004/libexec/lib/python3.10/site-packages/salt/cli/ssh.py", line 22, in run
    ssh.run()
  File "/opt/homebrew/Cellar/salt/3004/libexec/lib/python3.10/site-packages/salt/client/ssh/__init__.py", line 787, in run
    for ret in self.handle_ssh():
  File "/opt/homebrew/Cellar/salt/3004/libexec/lib/python3.10/site-packages/salt/client/ssh/__init__.py", line 606, in handle_ssh
    routine.start()
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread._local' object
Traceback (most recent call last):
  File "/opt/homebrew/bin/salt-ssh", line 8, in <module>
    sys.exit(salt_ssh())
  File "/opt/homebrew/Cellar/salt/3004/libexec/lib/python3.10/site-packages/salt/scripts.py", line 459, in salt_ssh
    client.run()
  File "/opt/homebrew/Cellar/salt/3004/libexec/lib/python3.10/site-packages/salt/cli/ssh.py", line 22, in run
    ssh.run()
  File "/opt/homebrew/Cellar/salt/3004/libexec/lib/python3.10/site-packages/salt/client/ssh/__init__.py", line 787, in run
    for ret in self.handle_ssh():
  File "/opt/homebrew/Cellar/salt/3004/libexec/lib/python3.10/site-packages/salt/client/ssh/__init__.py", line 606, in handle_ssh
    routine.start()
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/homebrew/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread._local' object

Setup

brew install saltstack on macOS Monterey

Saltfile

salt-ssh:
    config_dir: ./salt_root
    max_procs: 30
    wipe_ssh: true
    log_file: ./salt_root/log_ssh
    ssh_log_file: ./salt_root/log_ssh

salt_root/roster

strida-prod:
    host: 192.168.20.1
    user: root
    priv: ~/.ssh/id_ed25519

Steps to Reproduce the behavior

Attempt to use salt-ssh to connect to a remote host

Expected behavior

salt-ssh to run the test.ping command to validate a connection to the remote host.

Screenshots

See stack trace above.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
% env HOMEBREW_PREFIX=/opt/homebrew salt --versions-report
Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.0
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: 3.1.12
        Jinja2: 2.11.3
       libgit2: 1.3.0
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.19
      pycrypto: 3.11.0
  pycryptodome: 3.9.8
        pygit2: 1.7.0
        Python: 3.10.1 (main, Dec  6 2021, 22:18:13) [Clang 13.0.0 (clang-1300.0.29.3)]
  python-gnupg: 0.4.4
        PyYAML: 5.4.1
         PyZMQ: 21.0.2
         smmap: 3.0.2
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: darwin 21.2.0
        locale: utf-8
       machine: arm64
       release: 21.2.0
        system: Darwin
       version: 12.1 arm64

Activity

  1. lubyou commented on Jan 31, 2022

    @lubyou
    Contributor

    Hi,

    I ran into the same issue.

    I installed salt-ssh 3004 into a venv.
    This seems broken on when using python 3.8, 3.9 or 3.10 from brew.

    If I install salt-ssh from pip in a 3.7 venv, it seems to work.

    Working:

    Salt Version:
              Salt: 3004
    
    Dependency Versions:
              cffi: 1.14.6
          cherrypy: 18.6.1
          dateutil: 2.8.0
         docker-py: Not Installed
             gitdb: 4.0.5
         gitpython: 3.1.12
            Jinja2: 2.11.3
           libgit2: Not Installed
          M2Crypto: Not Installed
              Mako: 1.1.4
           msgpack: 1.0.2
      msgpack-pure: Not Installed
      mysql-python: Not Installed
         pycparser: 2.19
          pycrypto: Not Installed
      pycryptodome: 3.9.8
            pygit2: Not Installed
            Python: 3.7.12 (default, Oct 13 2021, 06:51:54)
      python-gnupg: 0.4.4
            PyYAML: 5.4.1
             PyZMQ: 18.0.1
             smmap: 3.0.2
           timelib: 0.2.4
           Tornado: 4.5.3
               ZMQ: 4.3.1
    
    System Versions:
              dist: darwin 19.6.0
            locale: UTF-8
           machine: x86_64
           release: 19.6.0
            system: Darwin
           version: 10.15.7 x86_64
    
  2. lubyou commented on Feb 1, 2022

    @lubyou
    Contributor

    Looks like the issue is due to the start method of the multiprocessing method being changed to spawn.

    When set to fork, the issue does not occur.

  3. CodingMinds commented on Apr 16, 2022

    @CodingMinds

    For others coming by via Google:

    Workaround to get it running with the latest homebrew:
    Modified /opt/homebrew/opt/salt/libexec/lib/python3.10/site-packages/salt/client/ssh/__init__.py and added multiprocessing.set_start_method("fork") as line 204.

    At least salt-ssh $HOSTNAME test.ping seems to work now.

  4. johnkeates commented on Aug 19, 2022

    @johnkeates

    Can confirm that this is still an issue on the latest OS, Python and salt-ssh packages as well as the fix still being switching the method to fork.

  5. garethgreenaway commented on Sep 27, 2022

    @garethgreenaway
    Contributor

    Anyone seeing this issue, can you please upgrade to the latest Salt release (3005) and see if the issue still occurs. Thanks!

  6. johnkeates commented on Sep 27, 2022

    @johnkeates

    This was solved on 3005 for me.

  7. garethgreenaway commented on Oct 4, 2022

    @garethgreenaway
    Contributor

    @johnkeates Thanks for the update. Since this one was fixed in the 3005 release I'll go ahead and close this one out, if the issue persists please feel free to open a new issue or request that this issue be reopened. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugbroken, incorrect, or confusing behaviorneeds-triage

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions