salt-run broken in 2015.8? #27130

githubcdr · 2015-09-15T12:03:15Z

Hi,

I noticed some strange errors using various salt-runners, it looks like salt-run;

 salt-run winrepo.update_git_repos
 salt-run manage.down -l all

[TRACE   ] data = Exception occurred in runner manage.down: Traceback (most recent call last):
Exception occurred in runner manage.down: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/salt/client/mixins.py", line 288, in low
    _functions = copy.deepcopy(self.functions)
  File "/usr/lib64/python2.6/copy.py", line 189, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib64/python2.6/copy.py", line 342, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib64/python2.6/copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python2.6/copy.py", line 255, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python2.6/copy.py", line 189, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib64/python2.6/copy.py", line 342, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib64/python2.6/copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python2.6/copy.py", line 255, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python2.6/copy.py", line 189, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib64/python2.6/copy.py", line 342, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib64/python2.6/copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python2.6/copy.py", line 255, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python2.6/copy.py", line 189, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib64/python2.6/copy.py", line 351, in _reconstruct
    y.__dict__.update(state)
AttributeError: 'NoneType' object has no attribute 'update'

I have this same bug confirmed on a Ubuntu 14.04 pip based Salstack installation.

Any help appreciated.

~~
Version information

CentOS6.7 patched

python --version
Python 2.6.6

[henkiewestbroek@saltstack salt ]# salt-master --versions-report
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.7.3
       M2Crypto: 0.20.2
           Mako: 1.0.0
         PyYAML: 3.11
          PyZMQ: 14.5.0
         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: 0.9.2
       cherrypy: 3.2.2
       dateutil: 2.4.0
          gitdb: 0.6.4
      gitpython: Not Installed
          ioflo: 1.1.2
        libnacl: 1.4.3
   msgpack-pure: 0.1.3
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.9.0
        timelib: 0.2.4

System Versions:
           dist: centos 6.7 Final
        machine: x86_64
        release: 2.6.32-573.3.1.el6.x86_64
         system: CentOS 6.7 Final

The text was updated successfully, but these errors were encountered:

jfindlay · 2015-09-15T17:17:58Z

@githubcdr, thanks for the report.

cachedout · 2015-09-15T22:09:56Z

I cannot replicate this at all. Can you provide additional information on how you installed your minion?

githubcdr · 2015-09-16T08:19:32Z

Installed from repo.saltstack.com

[root@saltstack states ]# rpm -qa | grep salt
salt-2015.8.0-2.el6.noarch
salt-cloud-2015.8.0-2.el6.noarch
salt-master-2015.8.0-2.el6.noarch
salt-minion-2015.8.0-2.el6.noarch
salt-api-2015.8.0-2.el6.noarch
salt-ssh-2015.8.0-2.el6.noarch

[root@saltstack states ]# cat /etc/yum.repos.d/saltstack.repo
[saltstack-repo]
name=SaltStack repo for RHEL/CentOS $releasever
baseurl=https://repo.saltstack.com/yum/rhel$releasever
enabled=1
gpgcheck=1
gpgkey=https://repo.saltstack.com/yum/rhel$releasever/SALTSTACK-GPG-KEY.pub

I completely purged salt from my system, even moved /var/salt/cache, but that didn't fix the problem.

Any suggestions on how to further debug this issue? Could it be related to the python installation itself? Or a configuration option?

(Will try to reproduce this in a container)

githubcdr · 2015-09-16T09:49:15Z

After a long docker debug-compare session I discovered that this issue seems related to the following master.conf setting;

cython_enable: True

After disabling this setting salt-run works again!

Really have no clue why how or what, but still feels this is a bug and might cause a lot of trouble for others in the same situation.

cachedout · 2015-09-17T18:25:58Z

I can't replicate this even with this option turned on. Do you have custom cython salt modules that you're using on your system?

githubcdr · 2015-09-23T12:47:13Z

I really have no idea, but I guess a RPM maintained version of cython vs a local pip installed cython caused this problem.

After removing the pip installed version and the RPM things worked again, reinstalling the Cython via yum caused this problem again.

I will leave this option disabled to prevent this issue.

DmitryKuzmenko · 2015-10-16T13:17:47Z

Just reproduced on Arch.

cachedout · 2015-10-16T13:19:46Z

@DmitryKuzmenko What was your installation method?

DmitryKuzmenko · 2015-10-16T13:32:01Z

@cachedout python 2.7 in virtualenv, salt in git repo fetched by tag v2015.8.0 and PYTHONPATH appended with ./

(venv-salt)$ ./scripts/salt --versions
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.10 (default, Sep  7 2015, 13:51:49)
           RAET: 0.6.1
        Tornado: 4.2.1
            ZMQ: 4.1.3
           cffi: Not Installed
       cherrypy: 3.8.0
       dateutil: 2.4.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: 1.3.9
        libnacl: 1.4.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist:   
        machine: x86_64
        release: 4.2.3-1-ARCH

cachedout · 2015-10-16T13:33:25Z

Wow, interesting. OK.

DmitryKuzmenko · 2015-10-16T13:34:01Z

@cachedout can I continue or you'd like to bring this up?

cachedout · 2015-10-16T13:34:40Z

I'm working on other things this morning but if you have time and want to investigate this, that would be great.

DmitryKuzmenko · 2015-10-16T13:37:03Z

I'll. Thank you.

DmitryKuzmenko · 2015-10-16T14:50:06Z

The issue appear because LazyLoader object imports module pyximport as it's field. Then the object is deepcopying that fails on copying the module object. As from copy module documentation, copy module isn't designed to copy module objects.

Fix: load pyximport into a global variable.

DmitryKuzmenko · 2015-10-16T16:40:07Z

@githubcdr thank you! You've caught two bugs at once.

githubcdr · 2015-10-16T16:46:59Z

@DmitryKuzmenko took some time to debug, but at least it was worth it

DmitryKuzmenko · 2015-10-19T14:17:20Z

I've done here. Found 2 more unneeded deepcopy. The second PR could be merged too.

jfindlay added this to the Approved milestone Sep 15, 2015

jfindlay added ZRELEASED - Beryllium ZRELEASED - Boron and removed ZRELEASED - Beryllium labels Sep 15, 2015

jfindlay added P3 Priority 3 and removed P2 Priority 2 labels Sep 17, 2015

cachedout added the Pending-Discussion The issue or pull request needs more discussion before it can be closed or merged label Sep 18, 2015

DmitryKuzmenko added the Confirmed Salt engineer has confirmed bug/feature - often including a MCVE label Oct 16, 2015

DmitryKuzmenko self-assigned this Oct 16, 2015

DmitryKuzmenko mentioned this issue Oct 16, 2015

LazyLoader deepcopy fix. #28020

Merged

DmitryKuzmenko added the fixed-pls-verify fix is linked, bug author to confirm fix label Oct 16, 2015

DmitryKuzmenko modified the milestones: B 10, Approved Oct 16, 2015

DmitryKuzmenko mentioned this issue Oct 16, 2015

Don't deepcopy just to iterate over dict keys/list items. #28035

Merged

DmitryKuzmenko mentioned this issue Oct 19, 2015

salt.wheel.WheelClient doesn't work follow the python api(AttributeError: 'NoneType' object has no attribute 'get') #26415

Closed

cachedout closed this as completed Nov 9, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

salt-run broken in 2015.8? #27130

salt-run broken in 2015.8? #27130

githubcdr commented Sep 15, 2015

jfindlay commented Sep 15, 2015

cachedout commented Sep 15, 2015

githubcdr commented Sep 16, 2015

githubcdr commented Sep 16, 2015

cachedout commented Sep 17, 2015

githubcdr commented Sep 23, 2015

DmitryKuzmenko commented Oct 16, 2015

cachedout commented Oct 16, 2015

DmitryKuzmenko commented Oct 16, 2015

cachedout commented Oct 16, 2015

DmitryKuzmenko commented Oct 16, 2015

cachedout commented Oct 16, 2015

DmitryKuzmenko commented Oct 16, 2015

DmitryKuzmenko commented Oct 16, 2015

DmitryKuzmenko commented Oct 16, 2015

githubcdr commented Oct 16, 2015

DmitryKuzmenko commented Oct 19, 2015

salt-run broken in 2015.8? #27130

salt-run broken in 2015.8? #27130

Comments

githubcdr commented Sep 15, 2015

jfindlay commented Sep 15, 2015

cachedout commented Sep 15, 2015

githubcdr commented Sep 16, 2015

githubcdr commented Sep 16, 2015

cachedout commented Sep 17, 2015

githubcdr commented Sep 23, 2015

DmitryKuzmenko commented Oct 16, 2015

cachedout commented Oct 16, 2015

DmitryKuzmenko commented Oct 16, 2015

cachedout commented Oct 16, 2015

DmitryKuzmenko commented Oct 16, 2015

cachedout commented Oct 16, 2015

DmitryKuzmenko commented Oct 16, 2015

DmitryKuzmenko commented Oct 16, 2015

DmitryKuzmenko commented Oct 16, 2015

githubcdr commented Oct 16, 2015

DmitryKuzmenko commented Oct 19, 2015