cloud.profile RuntimeError: dictionary changed size during iteration #24573

bailsman · 2015-06-10T17:17:37Z

Command I ran:

salt-run cloud.profile test_profile test_server

Exception:

Exception occurred in runner cloud.profile: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/client/mixins.py", line 335, in low
    data['return'] = self.functions[fun](*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/runners/cloud.py", line 106, in profile
    info = client.profile(prof, instances, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 336, in profile
    mapper = salt.cloud.Map(self._opts_defaults(**kwargs))
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 230, in _opts_defaults
    for provider in six.iterkeys(_providers):
RuntimeError: dictionary changed size during iteration

Version develop @9988310a5d75aab6da5b267acc8f38f1dd64f17e

Salt Version:
           Salt: 2015.5.2-5964-g9988310

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.3.1
         Python: 2.7.5 (default, Jun 17 2014, 18:11:42)
           RAET: Not Installed
            ZMQ: 3.2.5
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
       pycrypto: 2.6.1

System Versions:
           dist: centos 7.1.1503 Core
        machine: x86_64
        release: 3.10.0-123.8.1.el7.x86_64
         system: CentOS Linux 7.1.1503 Core

I fixed it locally with the below patch. (Not sure if correct)

--- a/salt/cloud/__init__.py
+++ b/salt/cloud/__init__.py
@@ -227,7 +227,7 @@ class CloudClient(object):
                          if a.get('provider', '')]
             if providers:
                 _providers = opts.get('providers', {})
-                for provider in six.iterkeys(_providers):
+                for provider in _providers.keys():
                     if provider not in providers:
                         _providers.pop(provider)
         return opts

Edit: the salt version here originally read 2015.5.0 because my .git folder was not accessible to the version detector. I fixed my version report above to report the version I tested with.

The text was updated successfully, but these errors were encountered:

basepi · 2015-06-11T23:31:21Z

That does look like it's probably the correct patch to me.

Thoughts, @rallytime or @techhat?

techhat · 2015-06-11T23:52:28Z

The blame on that line was @s0undt3ch. Perhaps he can offer insight.

s0undt3ch · 2015-07-24T12:42:15Z

We run that code in a copy so this is no longer an issue at least from 2015.5.1 onward...

jfindlay · 2015-07-25T04:02:16Z

@s0undt3ch, I am running into the same issue here on 2015.8 and develop.

rallytime · 2015-09-14T19:36:37Z

I am trying to reproduce this failure on 2015.8.0 using the EC2 driver and I am unable to do so. I've tried creating a minion with a regular salt-cloud command, using a runner, and using the module and state with a minion, like so:

# salt-cloud -p ec2-ubuntu-14 rallytime
# salt-run cloud.profile ec2-ubuntu-14 nt-u14-0
# salt rallytime cloud.profile ec2-ubuntu-14 nt-u14-1
# salt rallytime state.sls ec2-cloud-test

All of the minions were created successfully and return accordingly:

# salt '*' test.ping
rallytime:
    True
nt-u14-0:
    True
nt-u14-2:
    True
nt-u14-1:
    True

Versions report:

# salt --versions
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
         Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: 0.6.4
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.9.0
        timelib: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 4.1.5-x86_64-linode61
         system: Ubuntu 14.04 trusty

@codeHotter or @jfindlay Are you still hitting this issue? Is there something I am missing to try and reproduce this bug?

bailsman · 2015-09-15T10:17:20Z

@rallytime can still reproduce on develop and 2015.8

I think you need to have multiple providers configured to trigger the issue. I have two EC2 tenants configured in /etc/salt/cloud. If I comment one out, I can no longer trigger it.

rallytime · 2015-09-15T21:30:43Z

@codeHotter Can you post an example of what you're using to trigger this really quickly? (Sensitive data omitted, of course.)

bailsman · 2015-09-17T09:04:57Z

Not sure what information you need. Does this help?

# salt --versions-report
Salt Version:
           Salt: 2015.8.0-159-gfc8afcc

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.3.1
         Python: 2.7.5 (default, Jun 24 2015, 00:41:19)
           RAET: Not Installed
        Tornado: 4.2
            ZMQ: 3.2.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.1
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.1.1503 Core
        machine: x86_64
        release: 3.10.0-123.8.1.el7.x86_64
         system: CentOS Linux 7.1.1503 Core
# sed -e "/^\s*#/d; /^\s*$/d; s/\(id\|key\|master\|securitygroup\|keyname\): .*/\1: REDACTED/" /etc/salt/cloud
providers:
  ec2-ireland:
    minion:
      master: REDACTED
    ssh_interface: private_ips
    id: REDACTED
    key: REDACTED
    private_key: REDACTED
    keyname: REDACTED
    securitygroup: REDACTED
    location: eu-west-1
    availability_zone: eu-west-1b
    driver: ec2
  ec2-ireland-test:
    minion:
      master: REDACTED
    ssh_interface: public_ips
    id: REDACTED
    key: REDACTED
    private_key: REDACTED
    keyname: REDACTED
    securitygroup: REDACTED
    location: eu-west-1
    availability_zone: eu-west-1b
    driver: ec2
script: bootstrap-salt
# cat /etc/salt/cloud.profiles.d/test_profile.conf
test_profile:
  provider: ec2-ireland
  image: ami-e4ff5c93
  size: t2.micro
  deploy: false
root@mgmt ~/salt 2015-09-17 10:46:10
# salt-run cloud.profile test_profile my_profile_test
Exception occurred in runner cloud.profile: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/client/mixins.py", line 337, in lo                                                                                                                                   w
    data['return'] = self.functions[fun](*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/runners/cloud.py", line 106, in pr                                                                                                                                   ofile
    info = client.profile(prof, instances, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 337, in p                                                                                                                                   rofile
    mapper = salt.cloud.Map(self._opts_defaults(**kwargs))
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 231, in _                                                                                                                                   opts_defaults
    for provider in six.iterkeys(_providers):
RuntimeError: dictionary changed size during iteration

rallytime · 2015-09-17T17:11:49Z

That is prefect and very helpful. As you surmised earlier, you've found a very specific reproduction case! I think that you're correct in your fix originally posted. I tested both using salt-run and salt-cloud and both worked very well with the fix. I've submitted #27199 to clean up this issue.

Thanks again @codeHotter for your work on this. :)

basepi added Bug broken, incorrect, or confusing behavior severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around P2 Priority 2 Salt-Cloud labels Jun 11, 2015

basepi added this to the Approved milestone Jun 11, 2015

basepi added the help-wanted Community help is needed to resolve this label Jun 11, 2015

bailsman mentioned this issue Jul 23, 2015

cloud.profile state error: dictionary changed size during iteration #25641

Closed

s0undt3ch closed this as completed Jul 24, 2015

jfindlay added Confirmed Salt engineer has confirmed bug/feature - often including a MCVE RIoT Relates to integration with cloud providers, hypervisors, API-based services, etc. and removed help-wanted Community help is needed to resolve this labels Jul 25, 2015

jfindlay reopened this Jul 25, 2015

rallytime added the fixed-pls-verify fix is linked, bug author to confirm fix label Sep 14, 2015

rallytime removed the fixed-pls-verify fix is linked, bug author to confirm fix label Sep 15, 2015

rallytime modified the milestones: B 12, Approved Sep 17, 2015

rallytime self-assigned this Sep 17, 2015

rallytime mentioned this issue Sep 17, 2015

Avoid RunTimeError (dictionary changed size during iteration) with keys() #27199

Merged

rallytime added the fixed-pls-verify fix is linked, bug author to confirm fix label Sep 17, 2015

rallytime closed this as completed in 6b2a00e Sep 18, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cloud.profile RuntimeError: dictionary changed size during iteration #24573

cloud.profile RuntimeError: dictionary changed size during iteration #24573

bailsman commented Jun 10, 2015

basepi commented Jun 11, 2015

techhat commented Jun 11, 2015

s0undt3ch commented Jul 24, 2015

jfindlay commented Jul 25, 2015

rallytime commented Sep 14, 2015

bailsman commented Sep 15, 2015

rallytime commented Sep 15, 2015

bailsman commented Sep 17, 2015

rallytime commented Sep 17, 2015

cloud.profile RuntimeError: dictionary changed size during iteration #24573

cloud.profile RuntimeError: dictionary changed size during iteration #24573

Comments

bailsman commented Jun 10, 2015

basepi commented Jun 11, 2015

techhat commented Jun 11, 2015

s0undt3ch commented Jul 24, 2015

jfindlay commented Jul 25, 2015

rallytime commented Sep 14, 2015

bailsman commented Sep 15, 2015

rallytime commented Sep 15, 2015

bailsman commented Sep 17, 2015

rallytime commented Sep 17, 2015