[BUG] AttributeError: 'Process' object has no attribute '_args_for_getstate'

[shish]

Description
salt-ssh crashes with an exception before doing seemingly anything

Setup
Just upgraded to salt 3001, 3000 was working fine

Steps to Reproduce the behavior
salt-ssh $host state.highstate test=True

AttributeError: 'Process' object has no attribute '_args_for_getstate'

Traceback (most recent call last):
  File "/usr/local/bin/salt-ssh", line 33, in <module>
    sys.exit(load_entry_point('salt==3001', 'console_scripts', 'salt-ssh')())
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/scripts.py", line 499, in salt_ssh
    client.run()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/cli/ssh.py", line 26, in run
    ssh.run()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/client/ssh/__init__.py", line 787, in run
    for ret in self.handle_ssh():
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/client/ssh/__init__.py", line 604, in handle_ssh
    routine.start()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 755, in __getstate__
    args = self._args_for_getstate
AttributeError: 'Process' object has no attribute '_args_for_getstate'

Expected behavior
Salt executes on the host

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 3001

Dependency Versions:
           cffi: 1.12.2
       cherrypy: unknown
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 2.0.6
      gitpython: 2.1.15
         Jinja2: 2.10.1
        libgit2: 1.0.1
       M2Crypto: 0.35.2
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: Not Installed
   pycryptodome: 3.9.7
         pygit2: 1.2.1
         Python: 3.8.3 (default, May 27 2020, 20:54:22)
   python-gnupg: 0.4.4
         PyYAML: 5.1.2
          PyZMQ: 19.0.0
          smmap: 3.0.4
        timelib: 0.2.4
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: darwin 19.5.0
         locale: utf-8
        machine: x86_64
        release: 19.5.0
         system: Darwin
        version: 10.15.5 x86_64

[krionbsd]

@saltstack/team-core someone with MacOS Kung-Fu skills might take a look into this one

[cdalvaro]

Same for 'SignalHandlingProcess':

[salt.minion      :505 ][ERROR   ][308] Exception 'SignalHandlingProcess' object has no attribute '_args_for_getstate' occurred in scheduled job
[tornado.application:353 ][ERROR   ][308] Future <salt.ext.tornado.concurrent.Future object at 0x108ebf5b0> exception was never retrieved: Traceback (most recent call last):
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/minion.py", line 1719, in _handle_decoded_payload
    process.start()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 867, in start
    super(SignalHandlingProcess, self).start()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 755, in __getstate__
    args = self._args_for_getstate
AttributeError: 'SignalHandlingProcess' object has no attribute '_args_for_getstate'

Update: _args_for_getstate is only defined for Windows platforms:

salt/salt/utils/process.py

Lines 705 to 714 in 8a23ec9

	class Process(multiprocessing.Process, NewStyleClassMixIn):
	def __init__(self, *args, **kwargs):
	log_queue = kwargs.pop("log_queue", None)
	log_queue_level = kwargs.pop("log_queue_level", None)
	super(Process, self).__init__(*args, **kwargs)
	if salt.utils.platform.is_windows():
	# On Windows, subclasses should call super if they define
	# __setstate__ and/or __getstate__
	self._args_for_getstate = copy.copy(args)
	self._kwargs_for_getstate = copy.copy(kwargs)

So method __getstate__, should not been called from different architectures:

salt/salt/utils/process.py

Lines 745 to 766 in 8a23ec9

	# __setstate__ and __getstate__ are only used on Windows.
	def __setstate__(self, state):
	args = state["args"]
	kwargs = state["kwargs"]
	# This will invoke __init__ of the most derived class.
	self.__init__(*args, **kwargs)
	self._after_fork_methods = self._after_fork_methods
	self._finalize_methods = self._finalize_methods
	def __getstate__(self):
	args = self._args_for_getstate
	kwargs = self._kwargs_for_getstate
	if "log_queue" not in kwargs:
	kwargs["log_queue"] = self.log_queue
	if "log_queue_level" not in kwargs:
	kwargs["log_queue_level"] = self.log_queue_level
	return {
	"args": args,
	"kwargs": kwargs,
	"_after_fork_methods": self._after_fork_methods,
	"_finalize_methods": self._finalize_methods,
	}

Update 2: This part of the code has been untouched for more than 5 years, so the issue could be related with having changed from Python 3.7 to 3.8 on macOS with salt 3001 update Homebrew/homebrew-core#56461

[nlgordon]

I ran into this just running a test.ping against a mac minion. Getting my dev environment setup and going to take a swing at digging into this as time allows.

[cdalvaro]

This is a temporary solution until the bug is fixed.

Edit the Homebrew salt formula (brew edit salt) to apply the following patch:

diff --git a/Formula/salt.rb b/Formula/salt.rb
index d4a87977f..196e27648 100644
--- a/Formula/salt.rb
+++ b/Formula/salt.rb
@@ -17,7 +17,7 @@ class Salt < Formula
   depends_on "libgit2"
   depends_on "libyaml"
   depends_on "openssl@1.1"
-  depends_on "python@3.8"
+  depends_on "python"
   depends_on "zeromq"

   on_linux do
@@ -61,7 +61,7 @@ class Salt < Formula
     # https://github.com/Homebrew/homebrew-core/pull/52835#issuecomment-617502578
     File.write(buildpath/"pkg/osx/req_pyobjc.txt", "")

-    venv = virtualenv_create(libexec, Formula["python@3.8"].bin/"python3.8")
+    venv = virtualenv_create(libexec, "python3")
     venv.pip_install resources

     system libexec/"bin/pip", "install", "-v", "--ignore-installed", buildpath

then:

brew uninstall salt
sudo rm -rf /usr/local/Cellar/salt/3001
brew install --build-from-source salt

May be we could create a PR at homebrew-core to apply this patch to the formula until the next Salt released is ready with the bug fixed.

[sagetherage]

@weswhet

[cdalvaro]

I have tried adding this changes to salt/utils/process.py:

diff --git a/salt/utils/process.py b/salt/utils/process.py
index 2ac797b5f3..34e22b84cc 100644
--- a/salt/utils/process.py
+++ b/salt/utils/process.py
@@ -744,6 +744,9 @@ class Process(multiprocessing.Process, NewStyleClassMixIn):

     # __setstate__ and __getstate__ are only used on Windows.
     def __setstate__(self, state):
+        if not salt.utils.platform.is_windows():
+            self.__dict__.update(state)
+            return
         args = state["args"]
         kwargs = state["kwargs"]
         # This will invoke __init__ of the most derived class.
@@ -752,6 +755,8 @@ class Process(multiprocessing.Process, NewStyleClassMixIn):
         self._finalize_methods = self._finalize_methods

     def __getstate__(self):
+        if not salt.utils.platform.is_windows():
+            return self.__dict__
         args = self._args_for_getstate
         kwargs = self._kwargs_for_getstate
         if "log_queue" not in kwargs:

but then, the following error is reported:

[tornado.application:353 ][ERROR   ][1761] Future <salt.ext.tornado.concurrent.Future object at 0x1116c9550> exception was never retrieved: Traceback (most recent call last):
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/minion.py", line 1719, in _handle_decoded_payload
    process.start()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 872, in start
    super(SignalHandlingProcess, self).start()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Minion.setup_beacons.<locals>.handle_beacons'
[salt.minion      :505 ][ERROR   ][1761] Exception cannot pickle '_thread._local' object occurred in scheduled job

It looks this is the origin of the issue: https://docs.python.org/3/library/multiprocessing.html

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

Update

Similar issue solved here: horovod/horovod#1904

[weswhet]

Gonna need to put up the @s0undt3ch bat signal on this one. He’s already working on it here #57028

[weswhet]

Turning multiprocessing off in the minion config may patch this as well. If homebrew is going to be py3.8

[s0undt3ch]

This is a known issue being worked on in #57028 for magnesium

[sagetherage]

I will look at the PR and see if I can help in the test suite at all and look at if we can bring this into the point release.

[h0jeZvgoxFepBQ2C]

This is really a blocker somehow, using @cdalvaro 's solution worked fine on one laptop.. but not on the other.. Dunno why :(

[cdalvaro]

It is not working anymore because python formula has been upgraded from 3.7 to 3.8 ...

brew info python
python@3.8: stable 3.8.3 (bottled)
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@3.8/3.8.3_2 (4,229 files, 64.6MB)
  Poured from bottle on 2020-07-14 at 14:43:45
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.8.rb
==> Dependencies
Build: pkg-config ✔
Required: gdbm ✔, openssl@1.1 ✔, readline ✔, sqlite ✔, xz ✔
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python@3.8/libexec/bin

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.8/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Analytics
install: 463,840 (30 days), 1,253,709 (90 days), 1,905,378 (365 days)
install-on-request: 48,437 (30 days), 71,239 (90 days), 96,550 (365 days)
build-error: 0 (30 days)

We need to fix the code for making it to work with python 3.8

Applying the following patch to your salt formula (brew edit salt) and install it building from source will allow you to run salt-minion again.

diff --git a/Formula/salt.rb b/Formula/salt.rb
index d4a87977f..196e27648 100644
--- a/Formula/salt.rb
+++ b/Formula/salt.rb
@@ -17,7 +17,7 @@ class Salt < Formula
   depends_on "libgit2"
   depends_on "libyaml"
   depends_on "openssl@1.1"
-  depends_on "python@3.8"
+  depends_on "python@3.7"
   depends_on "zeromq"

   on_linux do
@@ -61,7 +61,7 @@ class Salt < Formula
     # https://github.com/Homebrew/homebrew-core/pull/52835#issuecomment-617502578
     File.write(buildpath/"pkg/osx/req_pyobjc.txt", "")

-    venv = virtualenv_create(libexec, Formula["python@3.8"].bin/"python3.8")
+    venv = virtualenv_create(libexec, Formula["python@3.7"].bin/"python3.7")
     venv.pip_install resources

     system libexec/"bin/pip", "install", "-v", "--ignore-installed", buildpath

brew install --build-from-source salt

Undo your formula changes after installing it to allow future updates for the salt formula.

[h0jeZvgoxFepBQ2C]

Do you have a workaround again maybe?

[sagetherage]

Looks like the PR was re-labeled as Silicon so I am matching it here. I see a change was requested so I will see if I can find out more details.

[prandelicious]

I'm getting the same error on Big Sur. Homebrew installs python 3.9.5 and the salt version is 3003.1

[nergdron]

yeah, just discovered this trying to help some coworkers with salt on their Macs. this is a big blocker for us, is there any update on getting it fixed?

[s0undt3ch]

I don't have much time available at the moment.
If you can, and want, to help, checkout the code from #57028
And try running the testsuite. It will fail, which means there are still places in Salt's code where we need fixes to support macOS python where spawning is now the default multiprocessing method.

[s0undt3ch]

I'll try to at least rebase the PR against the latest master in the next couple of days...