[BUG] AttributeError: 'Process' object has no attribute '_args_for_getstate'

[shish]

Description
salt-ssh crashes with an exception before doing seemingly anything

Setup
Just upgraded to salt 3001, 3000 was working fine

Steps to Reproduce the behavior
salt-ssh $host state.highstate test=True

AttributeError: 'Process' object has no attribute '_args_for_getstate'

Traceback (most recent call last):
  File "/usr/local/bin/salt-ssh", line 33, in <module>
    sys.exit(load_entry_point('salt==3001', 'console_scripts', 'salt-ssh')())
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/scripts.py", line 499, in salt_ssh
    client.run()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/cli/ssh.py", line 26, in run
    ssh.run()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/client/ssh/__init__.py", line 787, in run
    for ret in self.handle_ssh():
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/client/ssh/__init__.py", line 604, in handle_ssh
    routine.start()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 755, in __getstate__
    args = self._args_for_getstate
AttributeError: 'Process' object has no attribute '_args_for_getstate'

Expected behavior
Salt executes on the host

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 3001

Dependency Versions:
           cffi: 1.12.2
       cherrypy: unknown
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 2.0.6
      gitpython: 2.1.15
         Jinja2: 2.10.1
        libgit2: 1.0.1
       M2Crypto: 0.35.2
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: Not Installed
   pycryptodome: 3.9.7
         pygit2: 1.2.1
         Python: 3.8.3 (default, May 27 2020, 20:54:22)
   python-gnupg: 0.4.4
         PyYAML: 5.1.2
          PyZMQ: 19.0.0
          smmap: 3.0.4
        timelib: 0.2.4
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: darwin 19.5.0
         locale: utf-8
        machine: x86_64
        release: 19.5.0
         system: Darwin
        version: 10.15.5 x86_64

[krionbsd]

@saltstack/team-core someone with MacOS Kung-Fu skills might take a look into this one

[cdalvaro]

Same for 'SignalHandlingProcess':

[salt.minion      :505 ][ERROR   ][308] Exception 'SignalHandlingProcess' object has no attribute '_args_for_getstate' occurred in scheduled job
[tornado.application:353 ][ERROR   ][308] Future <salt.ext.tornado.concurrent.Future object at 0x108ebf5b0> exception was never retrieved: Traceback (most recent call last):
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/minion.py", line 1719, in _handle_decoded_payload
    process.start()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 867, in start
    super(SignalHandlingProcess, self).start()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 755, in __getstate__
    args = self._args_for_getstate
AttributeError: 'SignalHandlingProcess' object has no attribute '_args_for_getstate'

Update: _args_for_getstate is only defined for Windows platforms:

salt/salt/utils/process.py

Lines 705 to 714 in 8a23ec9

	class Process(multiprocessing.Process, NewStyleClassMixIn):
	def __init__(self, *args, **kwargs):
	log_queue = kwargs.pop("log_queue", None)
	log_queue_level = kwargs.pop("log_queue_level", None)
	super(Process, self).__init__(*args, **kwargs)
	if salt.utils.platform.is_windows():
	# On Windows, subclasses should call super if they define
	# __setstate__ and/or __getstate__
	self._args_for_getstate = copy.copy(args)
	self._kwargs_for_getstate = copy.copy(kwargs)

So method __getstate__, should not been called from different architectures:

salt/salt/utils/process.py

Lines 745 to 766 in 8a23ec9

	# __setstate__ and __getstate__ are only used on Windows.
	def __setstate__(self, state):
	args = state["args"]
	kwargs = state["kwargs"]
	# This will invoke __init__ of the most derived class.
	self.__init__(*args, **kwargs)
	self._after_fork_methods = self._after_fork_methods
	self._finalize_methods = self._finalize_methods
	def __getstate__(self):
	args = self._args_for_getstate
	kwargs = self._kwargs_for_getstate
	if "log_queue" not in kwargs:
	kwargs["log_queue"] = self.log_queue
	if "log_queue_level" not in kwargs:
	kwargs["log_queue_level"] = self.log_queue_level
	return {
	"args": args,
	"kwargs": kwargs,
	"_after_fork_methods": self._after_fork_methods,
	"_finalize_methods": self._finalize_methods,
	}

Update 2: This part of the code has been untouched for more than 5 years, so the issue could be related with having changed from Python 3.7 to 3.8 on macOS with salt 3001 update Homebrew/homebrew-core#56461

[nlgordon]

I ran into this just running a test.ping against a mac minion. Getting my dev environment setup and going to take a swing at digging into this as time allows.

[cdalvaro]

This is a temporary solution until the bug is fixed.

Edit the Homebrew salt formula (brew edit salt) to apply the following patch:

diff --git a/Formula/salt.rb b/Formula/salt.rb
index d4a87977f..196e27648 100644
--- a/Formula/salt.rb
+++ b/Formula/salt.rb
@@ -17,7 +17,7 @@ class Salt < Formula
   depends_on "libgit2"
   depends_on "libyaml"
   depends_on "openssl@1.1"
-  depends_on "python@3.8"
+  depends_on "python"
   depends_on "zeromq"

   on_linux do
@@ -61,7 +61,7 @@ class Salt < Formula
     # https://github.com/Homebrew/homebrew-core/pull/52835#issuecomment-617502578
     File.write(buildpath/"pkg/osx/req_pyobjc.txt", "")

-    venv = virtualenv_create(libexec, Formula["python@3.8"].bin/"python3.8")
+    venv = virtualenv_create(libexec, "python3")
     venv.pip_install resources

     system libexec/"bin/pip", "install", "-v", "--ignore-installed", buildpath

then:

brew uninstall salt
sudo rm -rf /usr/local/Cellar/salt/3001
brew install --build-from-source salt

May be we could create a PR at homebrew-core to apply this patch to the formula until the next Salt released is ready with the bug fixed.

[sagetherage]

@weswhet