Docker "AttributeError: 'RecentlyUsedContainer' object has no attribute 'lock'"

[strocknar]

I'm attempting to use salt.states.dockerio with a single "pulled" state, but keep getting the referenced error. I have tried updating docker-py and urllib3 (both through yum and pip), but to no avail. Any ideas as to where I might start looking to figure out what is causing this error?

-=# STATE #=-

centos:
    docker.pulled:
        - tag: latest

-=# ERROR #=-

      ID: centos
Function: docker.pulled
  Result: False
 Comment: An exception occurred in this state: Traceback (most recent call last):
            File "/usr/lib/python2.6/site-packages/salt/state.py", line 1529, in call
              **cdata['kwargs'])
            File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
              self.gen.throw(type, value, traceback)
            File "/usr/lib/python2.6/site-packages/salt/utils/context.py", line 41, in func_globals_inject
              yield
            File "/usr/lib/python2.6/site-packages/salt/state.py", line 1529, in call
              **cdata['kwargs'])
            File "/usr/lib/python2.6/site-packages/salt/states/dockerio.py", line 265, in pulled
              returned = pull(name, tag=tag)
            File "/usr/lib/python2.6/site-packages/salt/modules/dockerio.py", line 1588, in pull
              client = _get_client()
            File "/usr/lib/python2.6/site-packages/salt/modules/dockerio.py", line 273, in _get_client
              client._version = client.version()['ApiVersion']
            File "/usr/lib/python2.6/site-packages/docker/client.py", line 976, in version
            File "/usr/lib/python2.6/site-packages/docker/client.py", line 81, in _get
              try:
            File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 310, in get
              self.stream = False
            File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 279, in request
              __attrs__ = [
            File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 374, in send
              files=request.files,
            File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 155, in send
              **proxy_kwargs)
            File "/usr/lib/python2.6/site-packages/docker/unixconn/unixconn.py", line 78, in get_connection
              return pool
          AttributeError: 'RecentlyUsedContainer' object has no attribute 'lock'
 Started: 12:55:06.655070
Duration: 7.509 ms
 Changes:

[jfindlay]

@strocknar, what is the output of salt --versions-report or salt minion test.versions_report as necessary, and what version of docker are you using? Thanks.

[strocknar]

@jfindlay

-=# DOCKER VERSION #=-

$ sudo docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8/1.4.1
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8/1.4.1

-=# SALT VERSION #=-

$ sudo salt --versions-report
           Salt: 2014.7.1
         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
         Jinja2: 2.2.1
       M2Crypto: 0.20.2
 msgpack-python: 0.4.4
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 14.3.1
           RAET: Not Installed
            ZMQ: 3.2.4
           Mako: Not Installed

[strocknar]

Looks like this may be something that is fixed in 2014.7.2 and later. I'm using CentOS 6.6, and the packages in EPEL for docker and salt are out of date. Downloading the newest docker binary and doing 'pip install --upgrade salt==2014.7.2' seems to have cleared this (and brought about some other issues).

$ sudo salt --versions-report
           Salt: 2014.7.2
         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
         Jinja2: 2.7.3
       M2Crypto: 0.20.2
 msgpack-python: 0.4.6
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: Not Installed
          PyZMQ: 14.3.1
           RAET: Not Installed
            ZMQ: 3.2.4
           Mako: Not Installed

$ sudo docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef

[jfindlay]

@strocknar, yes, the docker module/state are known to need a cleanup. @terminalmage is currently working on this. Since the original issue has been resolved, do you mind if this issue is closed? Please open new issues if you find new bugs, though, or if you run into this issue again, we can reopen it, thanks.

[waynew]

I'm actually getting this identical issue, but with newer salt:

Minion versions report:

                      Salt: 2015.5.0
                    Python: 2.7.3 (default, Jun 22 2015, 19:33:41)
                    Jinja2: 2.6
                  M2Crypto: 0.21.1
            msgpack-python: 0.1.10
              msgpack-pure: Not Installed
                  pycrypto: 2.4.1
                   libnacl: Not Installed
                    PyYAML: 3.10
                     ioflo: Not Installed
                     PyZMQ: 14.0.1
                      RAET: Not Installed
                       ZMQ: 4.0.4
                      Mako: Not Installed
     Debian source package: 2015.5.0+ds-1precise1

Minion docker version:

    Client version: 1.7.1
    Client API version: 1.19
    Go version (client): go1.4.2
    Git commit (client): 786b29d
    OS/Arch (client): linux/amd64
    Server version: 1.7.1
    Server API version: 1.19
    Go version (server): go1.4.2
    Git commit (server): 786b29d
    OS/Arch (server): linux/amd64

Master:

           Salt: 2015.5.3
         Python: 2.7.10 (default, May 26 2015, 04:16:29)
         Jinja2: 2.8
       M2Crypto: 0.22
 msgpack-python: 0.4.6
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: Not Installed
          PyZMQ: 14.7.0
           RAET: Not Installed
            ZMQ: 4.1.2
           Mako: 1.0.1
        Tornado: 4.2.1

Stack trace:

   Function: docker.pulled
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1563, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/states/dockerio.py", line 271, in pulled
                  returned = pull(name, tag=tag, insecure_registry=insecure_registry)
                File "/usr/lib/python2.7/dist-packages/salt/modules/dockerio.py", line 1599, in pull
                  client = _get_client()
                File "/usr/lib/python2.7/dist-packages/salt/modules/dockerio.py", line 277, in _get_client
                  client._version = client.version()['ApiVersion']
                File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 837, in version
                  return self._result(self._get(url), json=True)
                File "/usr/local/lib/python2.7/dist-packages/docker/clientbase.py", line 86, in _get
                  return self.get(url, **self._set_request_timeout(kwargs))
                File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 310, in get
                  #: Stream response content default.
                File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 279, in request
                File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 374, in send
                  url=request.url,
                File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 155, in send
                  **proxy_kwargs)
                File "/usr/local/lib/python2.7/dist-packages/docker/unixconn/unixconn.py", line 71, in get_connection
                  with self.pools.lock:
              AttributeError: 'RecentlyUsedContainer' object has no attribute 'lock'
     Started: 23:38:03.795811
    Duration: 21.098 ms

[waynew]

I figured out the problem - in unixconn.py it's looking for the urllib3 that ships with requests first. However, I've got an older version of requests that doesn't have the newer version of urllib3, apparently.

[waynew]

As further information - after upgrading my versions I also had to restart the docker minion. It was holding on to the old urllib3 references and making things present very oddly.