Unhandled exception running status.ping_master

[bhenderson69]

Description of Issue

I am getting the following exception when testing minion failover and recovery between two salt-masters. The exception happens when the minion attempts to recover to the primary salt master server.

[INFO ] Running scheduled job: __master_alive_gsosalt2
[DEBUG ] LazyLoaded status.master
[DEBUG ] LazyLoaded status.ping_master
[DEBUG ] LazyLoaded status.master
[DEBUG ] LazyLoaded status.master
[DEBUG ] LazyLoaded status.ping_master
[DEBUG ] Initializing new AsyncAuth for ('c:\salt\conf\pki\minion', 'ATDSalt5.corp.qorvo.com', 'tcp://10.103.30.119:4506')
[DEBUG ] Subprocess SignalHandlingProcess-15-Schedule-__master_alive_gsosalt2 added
[DEBUG ] LazyLoaded config.get
**[ERROR ] Unhandled exception running status.ping_master
Traceback (most recent call last):
File "C:\salt\bin\lib\site-packages\salt-3002-py3.7.egg\salt\utils\schedule.py", line 817, in handle_func
ret["return"] = self.functions[func](*args, **kwargs)
File "C:\salt\bin\lib\site-packages\salt-3002-py3.7.egg\salt\modules\status.py", line 1736, in ping_master
opts = copy.deepcopy(opts)
NameError: name 'copy' is not defined
[DEBUG ] LazyLoaded config.get
[DEBUG ] LazyLoaded config.get
[ERROR ] Unhandled exception running status.ping_master
Traceback (most recent call last):
File "C:\salt\bin\lib\site-packages\salt-3002-py3.7.egg\salt\utils\schedule.py", line 817, in handle_func
ret["return"] = self.functions[func](*args, kwargs)
File "C:\salt\bin\lib\site-packages\salt-3002-py3.7.egg\salt\modules\status.py", line 1736, in ping_master
opts = copy.deepcopy(opts)
NameError: name 'copy' is not defined

Setup

Two salt-master servers are setup with on minion accepted.
I have configured the masters using multi-master PKI with failover as outlined in the following link:
https://docs.saltstack.com/en/3000/topics/tutorials/multimaster_pki.html

Steps to Reproduce Issue

1. Configure salt masters for multi-master PKI with Failover
2. Configure minion with both salt-masters and supporting settings:

master:

• gsosalt1
• gsosalt2
  verify_master_pubkey_sign: True
  master_type: failover
  master_alive_interval: 60
  master_failback_interval: 90

3. Allow minion to connect to primary.
4. Stop primary salt-master service
5. Let minion connect to secondary salt-master (verified salt-run manage.up)
6. Restart primary salt-master and wait for minion to fallback to primary...this is when the exception occurs.

Versions Report

Both masters:
Salt Version:
Salt: 3002.2

Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: 4.0.5
gitpython: 3.1.7
Jinja2: 2.11.2
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: 3.9.7
pygit2: Not Installed
Python: 3.8.6 (default, Sep 25 2020, 09:36:53)
python-gnupg: 0.4.6
PyYAML: 5.3.1
PyZMQ: 19.0.2
smmap: 3.0.4
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2

System Versions:
dist: ubuntu 20.10 groovy
locale: utf-8
machine: x86_64
release: 5.8.0-33-generic
system: Linux
version: Ubuntu 20.10 groovy

Minion:

Salt Version:
Salt: 3002

Dependency Versions:
           cffi: 1.12.2
       cherrypy: 17.4.1
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 2.0.5
      gitpython: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 1.0.0
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: Not Installed
   pycryptodome: 3.9.8
         pygit2: Not Installed
         Python: 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
   python-gnupg: 0.4.4
         PyYAML: 5.3.1
          PyZMQ: 18.0.1
          smmap: 2.0.5
        timelib: 0.2.4
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist:
         locale: cp1252
        machine: AMD64
        release: 10
         system: Windows
        version: 10 10.0.18362 SP0

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on Freenode
• SaltStack YouTube channel
• SaltStackInc Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at core@saltstack.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[sagetherage]

@krionbsd can you please triage this issue this week?

[sagetherage]

@bhenderson69 we don't yet support Ubuntu 20.10, are you only seeing it on this OS?

[bhenderson69]

Yes we are using the Ubuntu 20.10. Do you think this is an OS issue? I am seeing this on the client side which is Windows 10.
I tested again in my dev environment and here is the client side error:
DEBUG ] schedule: Job __master_failback was scheduled with a max number of 1
[INFO ] Running scheduled job: __master_failback
[DEBUG ] Subprocess SignalHandlingProcess-7-Schedule-__master_failback added
[DEBUG ] Multiprocessing queue logging configured for the process running under PID: 5940 at log level 10
[DEBUG ] schedule.handle_func: adding this job to the jobcache with data {'id': 'ATDSalt5.corp.qorvo.com', 'fun': 'status.ping_master', 'fun_args': [], 'schedule': '__master_failback', 'jid': '20210406202507729797', 'pid': 5940}
[DEBUG ] LazyLoaded status.ping_master
**[ERROR ] Unhandled exception running status.ping_master
Traceback (most recent call last):
File "C:\salt\bin\lib\site-packages\salt-3002-py3.7.egg\salt\utils\schedule.py", line 817, in handle_func
ret["return"] = self.functions[func](*args, kwargs)
File "C:\salt\bin\lib\site-packages\salt-3002-py3.7.egg\salt\modules\status.py", line 1736, in ping_master
opts = copy.deepcopy(opts)
NameError: name 'copy' is not defined
[DEBUG ] schedule.handle_func: Removing c:\salt\var\cache\salt\minion\proc\20210406202507729797
[DEBUG ] Subprocess SignalHandlingProcess-7-Schedule-__master_failback cleaned up
[DEBUG ] Validating beacon file
[DEBUG ] Not time to execute calibration beacons

[bhenderson69]

Also, I ran salt '*' status.ping_master localhost on the secondary master and got the following reply:

root@qfmsdev2:~# salt '*' status.ping_master localhost
qfmsdev2:
True
ATDSalt5.corp.qorvo.com:
The minion function caused an exception: Traceback (most recent call last):
File "C:\salt\bin\lib\site-packages\salt-3002-py3.7.egg\salt\minion.py", line 1889, in _thread_return
function_name, function_args, executors, opts, data
File "C:\salt\bin\lib\site-packages\salt-3002-py3.7.egg\salt\minion.py", line 1845, in _execute_job_function
return_data = self.executors[fname](opts, data, func, args, kwargs)
File "C:\salt\bin\lib\site-packages\salt-3002-py3.7.egg\salt\executors\direct_call.py", line 12, in execute
return func(*args, **kwargs)
File "C:\salt\bin\lib\site-packages\salt-3002-py3.7.egg\salt\modules\status.py", line 1736, in ping_master
opts = copy.deepcopy(opts)
NameError: name 'copy' is not defined
ERROR: Minions returned with non-zero exit code

I understand we are running a higher version of Ubuntu server for the master but it does feel like this is a server issue. Do you see this issue when running primary / failover configuration with Ubuntu 20.04? What is the client OS? Windows?

Regards,
Billy Henderson

[nschoonm]

+1, also experiencing this on windows minion

2022-10-05 17:26:16,601 [salt.utils.schedule][ERROR   ] Unhandled exception running status.ping_master
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.2-py3.8.egg\salt\utils\schedule.py", line 842, in handle_func
    ret["return"] = self.functions[func](*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.2-py3.8.egg\salt\loader\lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.2-py3.8.egg\salt\loader\lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.2-py3.8.egg\salt\loader\lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.2-py3.8.egg\salt\modules\status.py", line 1733, in ping_master
    opts = copy.deepcopy(__opts__)
NameError: name 'copy' is not defined