Intermittent push failures with 1.10.3

Using 1.10.3 with the Ansible docker_image module. Seeing roughly 4/5 pushes fail with errors like the following:

Error pushing image 127.0.0.1:5000/test-image-3: Extra data: line 2 column 1 - line 3 column 1 (char 64 - 128)

On the other 1/5 it works like a charm.

Seems like there is still a race condition as mentioned in #1059.

[shin-]

Seems odd that this would appear on 1.10.3, considering we merged #1081 - is it possible you have multiple versions of docker-py installed (through virtualenv or something) and ansible is using an older one?

@shin

I have virtualenv and virtualenvwrapper installed and I just re-did my laptop Friday to purge old virtual environments and my global space.

So I have 1 virtualenv with docker-py in it, global site-packages toggled off for that and no docker-py in my system environment.

@shin

This is from the virtualenv I am seeing the failures in.

pip freeze
-e git+https://github.com/ansible/ansible.git@07a76bece15d2568d7ea76e77266190652a0beec#egg=ansible
backports.ssl-match-hostname==3.5.0.1
cffi==1.8.2
cryptography==1.5
docker-py==1.10.3
docker-pycreds==0.2.1
enum34==1.1.6
idna==2.1
ipaddress==1.0.17
Jinja2==2.8
MarkupSafe==0.23
paramiko==2.0.2
py2-ipaddress==3.4.1
pyasn1==0.1.9
pycparser==2.14
pycrypto==2.6.1
PyYAML==3.12
requests==2.10.0
six==1.10.0
websocket-client==0.37.0

[shin-]

Ah, thanks for double-checking. It looks like #1081 didn't completely solve our issue, or maybe this is a different problem.

What version of the engine are you using?

Docker version 1.12.1, build 6f9534c

I am contemplating doing a fresh install of Python to avoid the El Capitan issues.

[shin-]

Would you mind installing docker-py from this source, and reporting here with the error it prints out?

pip install -e 'git+https://github.com/docker/docker-py.git@6b7a828400f46ea81374bc5764d8aa81bf38f6f7'

Hopefully it would help us identify what kind of data docker-py is choking on, and then we can write a patch to fix it.

Not a problem. I can do that.

Error

fatal: [test-image-3]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "api_version": null,
            "archive_path": null,
            "buildargs": null,
            "cacert_path": null,
            "cert_path": null,
            "container_limits": null,
            "debug": false,
            "docker_host": null,
            "dockerfile": null,
            "filter_logger": false,
            "force": false,
            "http_timeout": null,
            "key_path": null,
            "load_path": null,
            "name": "test-image-3",
            "nocache": "False",
            "path": null,
            "pull": false,
            "push": true,
            "repository": "127.0.0.1:5000/test-image-3",
            "rm": true,
            "ssl_version": null,
            "state": "present",
            "tag": "role-test",
            "timeout": null,
            "tls": null,
            "tls_hostname": null,
            "tls_verify": null,
            "use_tls": "no"
        },
        "module_name": "ilmn_docker_image"
    },
    "msg": "Error pushing image 127.0.0.1:5000/test-image-3: Extra data: line 2 column 1 - line 3 column 1 (char 64 - 128)"

packages

pip freeze
-e git+https://github.com/ansible/ansible.git@07a76bece15d2568d7ea76e77266190652a0beec#egg=ansible
backports.ssl-match-hostname==3.5.0.1
cffi==1.8.2
cryptography==1.5
-e git+https://github.com/docker/docker-py.git@6b7a828400f46ea81374bc5764d8aa81bf38f6f7#egg=docker_py
docker-pycreds==0.2.1
enum34==1.1.6
idna==2.1
ipaddress==1.0.17
Jinja2==2.8
MarkupSafe==0.23
paramiko==2.0.2
py2-ipaddress==3.4.1
pyasn1==0.1.9
pycparser==2.14
pycrypto==2.6.1
PyYAML==3.12
requests==2.10.0
six==1.10.0
websocket-client==0.37.0

Installed with

pip install -e 'git+https://github.com/docker/docker-py.git@6b7a828400f46ea81374bc5764d8aa81bf38f6f7#egg=docker-py'

[shin-]

That's interesting. Seems like the error might be in a different place than we originally suspected, since it wasn't caught here. Let me dig in some more.

OK. Let me know if you want me to do some more testing. Thanks.

[shin-]

Oh, I figured out why you're seeing the issue: https://github.com/ansible/ansible-modules-core/blob/devel/cloud/docker/docker_image.py#L428

Ansible does the decoding of data chunks itself, so it doesn't rely on our JSON parsing code, causing the issue at their level when the API sometimes sends multiple chunks at a time. I'm afraid this is something you'll have to report there, as there's little to be done on our end.

No problem. I will file a bug a against their module tonight. I work with them a lot so I should get a response. Thanks for digging.

[shin-]

Cool. Thanks for helping me diagnose this. :)

I'll close the issue, but feel free to comment or reopen if there's more info or if you have other questions.