Skip to content
This repository has been archived by the owner on Oct 30, 2018. It is now read-only.

docker pull broken in devel for docker >= 1.8 #2043

Closed
jfindley opened this issue Sep 10, 2015 · 3 comments
Closed

docker pull broken in devel for docker >= 1.8 #2043

jfindley opened this issue Sep 10, 2015 · 3 comments

Comments

@jfindley
Copy link

Issue type

Bug report

Summary

The docker module fails when pulling new images when using docker >= 1.8.

Ansible version:

ansible 2.0.0 (detached HEAD 90e005d234) last updated 2015/09/10 09:32:44 (GMT +100)
  lib/ansible/modules/core: (devel 1c7e8ef1a0) last updated 2015/09/09 17:21:32 (GMT +100)
  lib/ansible/modules/extras: (detached HEAD ff2386faf4) last updated 2015/07/21 18:18:28 (GMT +100)
  config file = /etc/ansible/ansible.cfg
  configured module search path = None

Environment/config:

OSX source install. Default config.

Steps to reproduce

Add a docker image that does not exist on the target host.
Ansible will mark the pull as failed, even though it completes successfully. Log below:

"{\"status\":\"Status: Downloaded newer image for docker.io/mesoscloud/mesos-master:0.23.0-centos-7\"}\r\n", "{}\r\n"], 
"failed": true, "msg": "Unrecognized status from pull.", "status": ""}
fatal: [172.21.22.225]: FAILED! => {"changed": false, "changes": ["{\"stream\":\"Trying to pull repository docker.io/mesoscloud/mesos-master ... \"}\r\n", 
"{\"status\":\"Pulling from mesoscloud/mesos-master\",\"id\":\"0.23.0-centos-7\"}\r\n", 

... Snipped lots of repeating progress messages  ...

"{\"status\":\"Extracting\",\"progressDetail\":{\"current\":32,\"total\":32},\"progress\":\"[==================================================\\u003e]     32 B/32 B\",\"id\":\"db34bb6f738d\"}", 
"{\"status\":\"Extracting\",\"progressDetail\":{\"current\":32,\"total\":32},\"progress\":\"[==================================================\\u003e]     32 B/32 B\",\"id\":\"db34bb6f738d\"}", "{\"status\":\"Pull complete\",\"progressDetail\":{},\"id\":\"db34bb6f738d\"}", "{\"status\":\"Digest: sha256:3cf66154ed913a72cb4d49254f45b4a0efa75aadc7aa98d04bc3344e9cfd1a19\"}\r\n",
 "{\"status\":\"Status: Downloaded newer image for docker.io/mesoscloud/mesos-master:0.23.0-centos-7\"}\r\n", "{}\r\n"], 
"failed": true, "msg": "Unrecognized status from pull.", "status": ""}

Note that the docker status is "Pull complete" and that the image exists in sudo docker show images after this. It appears that ansible does not correctly parse the status message.

This only happens for me after upgrading from docker 1.7 to 1.8.

Re-running ansible works correctly, as now the image is downloaded and the pull is not required - however having to run ansible repeatedly to download multiple images is highly frustrating.

@AdamWill
Copy link

AdamWill commented Oct 9, 2015

I'm seeing this too, with docker-1.8.2-5.gitcb216be.fc23.x86_64 , python-docker-py-1.3.1-1.fc23.noarch (on the docker host box) and ansible-1.9.3-3.fc23.noarch (on the machine running ansible).

@AdamWill
Copy link

Note: the problem here isn't that the status message has changed, I don't think. Note in the error message, it actually logs the status it got: "status": "" - the value is the empty string. So the problem is that it's not actually getting the status properly, somehow.

The code here is around https://github.com/AdamWill/ansible-modules-core/blob/devel/cloud/docker/docker.py#L1392 . It runs a self.client.pull ; self.client is a docker.Client instance, so there we're in https://github.com/docker/docker-py/blob/master/docker/api/image.py#L150 . It saves the result as a list and then pulls the last item out of the list, and treats it as JSON, and gets the status value out of that JSON.

Now I look at the output carefully it's pretty obvious what the problem is: the very last item in the stream is an empty dict! Look at the end:

,"{\"status\":\"Status: Downloaded newer image for docker.io/mesoscloud/mesos-master:0.23.0-centos-7\"}\r\n", "{}\r\n"]

that second-to-last dict is the one we actually want - the one with the status message that the code is looking for - but there's a completely empty dict after it. So we're getting that empty dict, and obviously hence our status value will be the empty string.

So I guess the real problem here is in either docker-py or docker itself - whatever's causing that extra empty dict to show up in the stream. A dumb hack would be to take the last item, see if it's an empty dict, and take the second-to-last item if so, but that seems kinda awful. Having fun trying to track down what's causing this, though.

abadger added a commit that referenced this issue Oct 14, 2015
fix #2043: strip empty dict from end of 'pull' stream
abadger pushed a commit that referenced this issue Oct 14, 2015
When pulling an image using Docker 1.8, it seems the output
JSON stream has an empty dict at the very end. This causes
ansible to fail when pulling an image, as it's expecting a
status message in that dict which it uses to determine whether
it had to download the image or not. As a bit of an ugly hack
for that which remains backward compatible, try the last item
in the stream, and if it's an empty dict, take the last-but-one
item instead.

The strip() is needed as the exact value appears to be '{}/r/n';
we could just match that, but it seems like the kind of thing
where maybe it'd happen to just be '{}/n' or '{}' or something
in some cases, so let's just use strip() in case.
@AdamWill
Copy link

For the record, I filed the underlying issue upstream with Docker:

moby/moby#17031

so if that ever gets resolved, this hack could be retired after a suitable grace period for people to update Docker...

tourunen added a commit to CSCfi/pebbles that referenced this issue Nov 24, 2015
- see ansible/ansible-modules-core#2043
- images pulled with cli commands before launching containers
- images defined as variables
tourunen added a commit to CSCfi/pebbles that referenced this issue Dec 19, 2015
- see ansible/ansible-modules-core#2043
- images pulled with cli commands before launching containers
- images defined as variables
Jyrsa pushed a commit to CSCfi/pebbles-deploy that referenced this issue Jan 27, 2017
- see ansible/ansible-modules-core#2043
- images pulled with cli commands before launching containers
- images defined as variables
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants