"RuntimeError: maximum recursion depth exceeded" in salt/utils/lazy.py, using Salt-SSH #32591

AndreiPashkin · 2016-04-14T18:11:54Z

Description of Issue/Question

I got such traceback, calling such command sudo salt-ssh -ldebug -w "*some-expr*" state.apply test=True pillarenv='my_env' saltenv='my_env':

[ERROR   ] JSON Render failed for: 
Traceback (most recent call last):
  File "/tmp/.ubuntu_1804ae_salt/salt-call", line 15, in <module>
    salt_call()
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/scripts.py", line 333, in salt_call
    client.run()
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/cli/call.py", line 58, in run
    caller.run()
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/cli/caller.py", line 133, in run
    ret = self.call()
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/cli/caller.py", line 196, in call
    ret['return'] = func(*args, **kwargs)
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/modules/state.py", line 1456, in pkg
    ret = st_.call_chunks(lowstate)
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/state.py", line 1769, in call_chunks
    running = self.call_chunk(low, running, chunks)
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/state.py", line 2030, in call_chunk
    running = self.call_chunk(chunk, running, chunks)
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/state.py", line 2041, in call_chunk
    running = self.call_chunk(low, running, chunks)
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/state.py", line 2041, in call_chunk
    running = self.call_chunk(low, running, chunks)

...

  File "/tmp/.ubuntu_1804ae_salt/py2/salt/state.py", line 2041, in call_chunk
    running = self.call_chunk(low, running, chunks)
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/state.py", line 2041, in call_chunk
    running = self.call_chunk(low, running, chunks)
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/state.py", line 1938, in call_chunk
    low = self._mod_aggregate(low, running, chunks)
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/state.py", line 735, in _mod_aggregate
    agg_opt = self.functions['config.option']('state_aggregate')
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/loader.py", line 1052, in __getitem__
    func = super(LazyLoader, self).__getitem__(item)
  File "/tmp/.ubuntu_1804ae_salt/py2/salt/utils/lazy.py", line 83, in __getitem__
    if self._missing(key):
RuntimeError: maximum recursion depth exceeded
[ERROR   ] No JSON object could be decoded

I have two such states:

...

run_migrations:
  cmd.run:
    - name: run-migrations-script
    - user: {{ pillar['user'] }}
    - stateful:
      - test_name: run-migrations-script test
    - require:
      - file: some_file
      - virtualenv: some_virtualenv
      - postgres_user: some_postgres_user
      - postgres_database: some_database

db_backup:
  cmd.run:
    - name: pg_dump <options>
    - user: {{ pillar['user'] }}
    - env:
      - PGPASSWORD: {{ pillar['postgres_db_superuser_password'] }}
    - prereq:
      - cmd: run_migrations

...

When I remove prepreq - everyting runs normally.

Versions Report

$sudo salt-ssh --versions-report

Salt Version:
Salt: 2016.3.0rc2

Dependency Versions:
Jinja2: 2.7.2
M2Crypto: 0.21.1
Mako: 0.9.1
PyYAML: 3.10
PyZMQ: 14.0.1
Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
RAET: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.4
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.4.2
gitdb: 0.5.4
gitpython: 0.3.2 RC1
ioflo: Not Installed
libgit2: Not Installed
libnacl: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.3.0
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
python-gnupg: Not Installed
smmap: 0.8.2
timelib: Not Installed

System Versions:
dist: LinuxMint 17.3 rosa
machine: x86_64
release: 3.19.0-32-generic
system: Linux
version: LinuxMint 17.3 rosa

The text was updated successfully, but these errors were encountered:

jfindlay · 2016-04-15T16:08:09Z

@AndrewPashkin, thanks for reporting.

cachedout · 2016-04-18T17:44:33Z

I can't seem to reproduce this. It does look like your prereq references a different state than the one you posted, though. Could you reduce this to a minimal test case?

AndreiPashkin · 2016-04-18T17:46:09Z

@cachedout
I will, when I have free time.

cachedout · 2016-04-18T17:46:47Z

@AndrewPashkin Many thanks.

AndreiPashkin · 2016-04-23T17:26:06Z

@cachedout
I managed to reproduce the error with such SLS and Salt-SSH (didn't tried other execution methods):

foo:
  file.managed:
    - name: /tmp/foo
    # If remove this, the error goes away
    - unless: true

state1:
  cmd.run:
    - name: echo changed=yes
    - stateful:
      - test_name: echo changed=yes
    - require:
      - file: foo

hello:
  cmd.run:
    - name: echo hello
    - prereq:
      - cmd: state1

$ salt-ssh --versions-report
Salt Version:
           Salt: 2016.3.0-347-g13d2129

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: Not Installed
           Mako: 1.0.4
         PyYAML: 3.11
          PyZMQ: 15.2.0
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.2
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: LinuxMint 17.3 rosa
        machine: x86_64
        release: 3.19.0-32-generic
         system: Linux
        version: LinuxMint 17.3 rosa

thatch45 · 2016-07-20T22:23:55Z

Ok, this is a type checking issue, the root of the problem is that the value passed to unless resolves as a bool and we were not checking for it before. The above PR fixes this issue for me

alexforster · 2017-02-07T17:00:20Z

Did #34838 make it into 2016.11? I've just encountered this issue, and I only have a single prereq in my state–

reset-networking:
  cmd.run:
    - name: |
        for iface in $(find /sys/class/net -maxdepth 1 -type l -printf "%f\n") ; do
          ip route flush dev $iface;
          ip addr flush $iface;
          ifdown --force $iface;
        done
    - order: last
    - prereq:
      - module: apply-networking

apply-networking:
  module.run:
    - name: ip.apply_network_settings
    - order: last
    - onchanges:
      - file: /etc/network/interfaces

    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1412, in _thread_return
        return_data = executor.execute()
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 28, in execute
        return self.func(*self.args, **self.kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 816, in highstate
        orchestration_jid=orchestration_jid)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 3458, in call_highstate
        return self.state.call_high(high, orchestration_jid)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2298, in call_high
        ret = dict(list(disabled.items()) + list(self.call_chunks(chunks).items()))
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1814, in call_chunks
        running = self.call_chunk(low, running, chunks)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2099, in call_chunk
        running = self.call_chunk(low, running, chunks)
...
...
...
...
...
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2099, in call_chunk
        running = self.call_chunk(low, running, chunks)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1996, in call_chunk
        low = self._mod_aggregate(low, running, chunks)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 762, in _mod_aggregate
        agg_opt = self.functions['config.option']('state_aggregate')
      File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1086, in __getitem__
        func = super(LazyLoader, self).__getitem__(item)
      File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 91, in __getitem__
        if self._missing(key):
    RuntimeError: maximum recursion depth exceeded

root@linspector1:~# salt-call --versions-report
Salt Version:
           Salt: 2016.11.2

Dependency Versions:
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.9 (default, Jun 29 2016, 13:08:31)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: debian 8.7
        machine: x86_64
        release: 3.16.0-4-amd64
         system: Linux
        version: debian 8.7

rmader · 2017-08-30T08:51:43Z

I also just encountered the issue. I'm using tons of prereqs, but will try to create a test

Edit: ok turned out to be a logical failure of mine. The poisonous part was:

create_link_{{ volume.get('name') }}:
  file.symlink:
    - name: /var/lib/lxd/storage-pools/{{ volume.get('name') }}
    - target: {{ mount_dir }}/{{ volume.get('name') }}
    - prereq_in:
      - file: remove_folder_{{ volume.get('name') }}
      
remove_folder_{{ volume.get('name') }}:
  file.absent:
    - name: /var/lib/lxd/storage-pools/{{ volume.get('name') }}
    - prereq_in:
      - mount: unmount_{{ volume.get('name') }}

unmount_{{ volume.get('name') }}:
  mount.unmounted:
    - name: /var/lib/lxd/storage-pools/{{ volume.get('name') }}

When changed to the following, it worked again:

create_link_{{ volume.get('name') }}:
  file.symlink:
    - name: /var/lib/lxd/storage-pools/{{ volume.get('name') }}
    - target: {{ mount_dir }}/{{ volume.get('name') }}
    - force: True
    - prereq_in:
      - mount: unmount_{{ volume.get('name') }}

unmount_{{ volume.get('name') }}:
  mount.unmounted:
    - name: /var/lib/lxd/storage-pools/{{ volume.get('name') }}

The minion function caused an exception: Traceback (most recent call last):
                    File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1466, in _thread_return
                      return_data = executor.execute()
                    File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 28, in execute
                      return self.func(*self.args, **self.kwargs)
                    File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 1109, in sls
                      ret = st_.state.call_high(high_, orchestration_jid)
                    File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2515, in call_high
                      ret = self.call_chunks(chunks)
                    File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2008, in call_chunks
                      running = self.call_chunk(low, running, chunks)
                    File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2329, in call_chunk
                      running = self.call_chunk(chunk, running, chunks)
                    File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2329, in call_chunk
                      running = self.call_chunk(chunk, running, chunks)
                    File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2340, in call_chunk
                      running = self.call_chunk(low, running, chunks)
                    File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2340, in call_chunk
                      running = self.call_chunk(low, running, chunks)
                    File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2340, in call_chunk
                      running = self.call_chunk(low, running, chunks)
                    
                   ....

                    File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2237, in call_chunk
                      low = self._mod_aggregate(low, running, chunks)
                    File "/usr/lib/python2.7/dist-packages/salt/state.py", line 768, in _mod_aggregate
                      agg_opt = self.functions['config.option']('state_aggregate')
                    File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1113, in __getitem__
                      func = super(LazyLoader, self).__getitem__(item)
                    File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 91, in __getitem__
                      if self._missing(key):
                  RuntimeError: maximum recursion depth exceeded

salt-call --versions-report
Salt Version:
           Salt: 2017.7.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-92-generic
         system: Linux
        version: Ubuntu 16.04 xenial

jfindlay added Bug broken, incorrect, or confusing behavior severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around Core relates to code central or existential to Salt Salt-SSH P2 Priority 2 labels Apr 15, 2016

jfindlay added this to the Approved milestone Apr 15, 2016

daveneeley mentioned this issue May 28, 2016

RuntimeError maximum recursion depth exceeded in config.option state_aggregate #33598

Closed

thatch45 mentioned this issue Jul 20, 2016

Check if a valid value is passed to unlyif/unless #34838

Merged

rallytime added the fixed-pls-verify fix is linked, bug author to confirm fix label Jul 20, 2016

rallytime closed this as completed Aug 30, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"RuntimeError: maximum recursion depth exceeded" in salt/utils/lazy.py, using Salt-SSH #32591

"RuntimeError: maximum recursion depth exceeded" in salt/utils/lazy.py, using Salt-SSH #32591

AndreiPashkin commented Apr 14, 2016 •

edited

Loading

jfindlay commented Apr 15, 2016

cachedout commented Apr 18, 2016

AndreiPashkin commented Apr 18, 2016

cachedout commented Apr 18, 2016

AndreiPashkin commented Apr 23, 2016 •

edited

Loading

thatch45 commented Jul 20, 2016

alexforster commented Feb 7, 2017

rmader commented Aug 30, 2017 •

edited

Loading

"RuntimeError: maximum recursion depth exceeded" in salt/utils/lazy.py, using Salt-SSH #32591

"RuntimeError: maximum recursion depth exceeded" in salt/utils/lazy.py, using Salt-SSH #32591

Comments

AndreiPashkin commented Apr 14, 2016 • edited Loading

Description of Issue/Question

Versions Report

jfindlay commented Apr 15, 2016

cachedout commented Apr 18, 2016

AndreiPashkin commented Apr 18, 2016

cachedout commented Apr 18, 2016

AndreiPashkin commented Apr 23, 2016 • edited Loading

thatch45 commented Jul 20, 2016

alexforster commented Feb 7, 2017

rmader commented Aug 30, 2017 • edited Loading

AndreiPashkin commented Apr 14, 2016 •

edited

Loading

AndreiPashkin commented Apr 23, 2016 •

edited

Loading

rmader commented Aug 30, 2017 •

edited

Loading