error while using docker for mac client 2.5.0.0 #2696

mattmoskwa · 2020-11-02T20:05:43Z

Using python 3.8.5 and docker-py version 4.3.1

  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/docker/models/containers.py", line 887, in get
    resp = self.client.api.inspect_container(container_id)
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/docker/api/container.py", line 771, in inspect_container
    self._get(self._url("/containers/{0}/json", container)), True
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/docker/api/client.py", line 228, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/requests/sessions.py", line 685, in send
    r.content
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/requests/models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/Users/mmoskwa/.local/share/virtualenvs/lunchbox-5JMc0Mx2/lib/python3.8/site-packages/requests/models.py", line 754, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

mconcas · 2020-11-03T11:17:45Z

Hello, same/similar issue.

Python version: 3.8.6
Docker version: 2.5.0
docker-py version: 4.3.1
urllib3 version: 1.25.11

reproducer:

import docker

client = docker.from_env()
client.containers.get("<non-existent-container-name>")

expected result:
Exception at docker api level (docker.errors.NotFound or docker.errors.APIError)

result:

Traceback (most recent call last):
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 696, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 436, in _error_catcher
    yield
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 763, in read_chunked
    self._update_chunk_length()
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 700, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/models.py", line 751, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 571, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 792, in read_chunked
    self._original_response.close()
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "reproducer.py", line 5, in <module>
    client.containers.get("asdrubale")
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/models/containers.py", line 887, in get
    resp = self.client.api.inspect_container(container_id)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/api/container.py", line 771, in inspect_container
    self._get(self._url("/containers/{0}/json", container)), True
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/api/client.py", line 228, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/sessions.py", line 685, in send
    r.content
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/models.py", line 754, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))```

dkleissa · 2020-11-03T16:46:28Z

An interesting observation on various version. I'm seeing this same issue on:

latest Docker Desktop for Mac stable channel (2.5.0.0)
latest Docker Desktop for Windows edge channel (2.4.2.0) on HyperV

I do not see this issue on:

latest Docker Desktop for Windows stable channel (2.5.0.0) on WSL2
latest Docker Desktop for Windows stable channel (2.5.0.0) on HyperV

dkleissa · 2020-11-03T19:11:31Z

We use this pattern to check if a container exists already before starting a container or when fetching the status of a container, as our software uses deterministic container names. In the case where the container isn't running we catch docker.errors.NotFound, which is now broken as @mconcas points out. After initial testing, it does appear this is the only place in the API that is broken like this.

A very temporary fix is catching the unexpected error, again this is unexpected, fragile, and possibly not useful for everyone's use case:

import docker

client = docker.from_env()
try:
    client.containers.get("<non-existent-container-name>")
except docker.errors.NotFound:
    pass
except requests.exceptions.ChunkedEncodingError:
    pass

reynoldsdj · 2020-11-04T17:31:32Z

We use this pattern to check if a container exists already before starting a container or when fetching the status of a container, as our software uses deterministic container names. In the case where the container isn't running we catch docker.errors.NotFound, which is now broken as @mconcas points out. After initial testing, it does appear this is the only place in the API that is broken like this.

A very temporary fix is catching the unexpected error, again this is unexpected, fragile, and possibly not useful for everyone's use case:
import docker

client = docker.from_env()
try:
    client.containers.get("<non-existent-container-name>")
except docker.errors.NotFound:
    pass
except requests.exceptions.ChunkedEncodingError:
    pass

I have the same exact problem, and I implemented the same fix. I just came here to open an issue. :)

Python 3.7.9 / docker-py 4.3.1 / Docker:

tophers42 · 2020-11-05T01:30:47Z

Seeing the same issue using wsl1 on windows:

Python 3.8.2
Linux 5.4.39-linuxkit x86_64
Docker version 18.09.1, build 4c52b90
docker-py 4.3.1

kevinkwoncareem · 2020-11-06T13:12:42Z

Mac facing the same problem :) hope it gets resolved! thanks guys

mconcas · 2020-11-09T16:01:27Z

Also on Linux:

Server Version: 19.03.13

Python 3.6.8 (default, Apr  2 2020, 13:34:55)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
>>> client = docker.from_env()
>>> client.containers.get("<non-existent-container-name>")
Traceback (most recent call last):
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status
    response.raise_for_status()
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.40/containers/%3Cnon-existent-container-name%3E/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/models/containers.py", line 887, in get
    resp = self.client.api.inspect_container(container_id)
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/api/container.py", line 771, in inspect_container
    self._get(self._url("/containers/{0}/json", container)), True
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/api/client.py", line 265, in _result
    self._raise_for_status(response)
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/mconcas/.virtualenvs/alidock/lib64/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("No such container: <non-existent-container-name>")

avirshup · 2020-11-10T01:06:08Z

Same issue - I was able to downgrade to 2.4 (link) to get it working again, no factory reset necessary.

yannickperrenet · 2020-11-10T12:59:51Z

They have just released version 2.5.0.1 which states to have fixed another Mac related issue (as you can see here in their docs). Not sure whether that also fixes this issue (which I sadly cannot test as I do not have a Mac), but I figured it was worth mentioning.

reynoldsdj · 2020-11-10T13:35:01Z

Docker 2.5.0.1 does not fix this issue on the mac. I upgraded to 2.5.0.1 and tested it. 2.5.0.1 fixes an issue with Big Sur and Virtualbox.

jukkei · 2020-11-10T13:40:29Z

Will test it soon. Somehow new docker-desktop downgraded kubernetes to 1.16.6-beta.0 from 1.19.3 (by using kubectl get nodes), but that's not related to this issue.

ch3ck · 2020-11-12T23:13:09Z

This issue is affecting our container builds, I don't know if there's a release coming out soon to fix this.

mconcas · 2020-11-13T11:20:09Z

As a side note: the CI does not catch the problem because the default docker version in Travis is still 18.x ...
This will be true even if they update from xenial to bionic and will require further action on the CI config.
Moreover I could not find the unit test that checks for this.

fixes docker/docker-py#2696 by catching exceptions.

thaJeztah · 2020-11-18T11:56:05Z

I think this may be the same issue as docker/for-mac#5025 / docker/for-mac#5058 / docker/for-mac#5059. More details can be found in this comment: docker/for-mac#5059 (comment)

thaJeztah · 2020-11-18T11:58:13Z

If the python code you're running is running on the mac itself, I'm not sure if there's a workaround, but if the code is running inside a container, you can (as a workaround) bind-mount /var/run/docker.sock.raw instead of /var/run/docker.sock to skip the docker desktop API proxy

thaJeztah · 2020-11-18T12:07:02Z

Oh, actually, there might be a workaround; the "raw" socket is also available on the mac side in;

$ ls -la ~/Library/Containers/com.docker.docker/Data/docker.raw.sock

srwxr-xr-x   1 sebastiaan  staff         0 Nov 16 11:24 docker.raw.sock=

e.g.;

$ docker -H unix:///Users/sebastiaan/Library/Containers/com.docker.docker/Data/docker.raw.sock version
Client: Docker Engine - Community
 Version:           20.10.0-beta1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        ac365d7
 Built:             Tue Oct 13 18:13:53 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true
...

⚠️ ⚠️ BIG FAT WARNING!!! ⚠️ ⚠️

I think the name of that socket will be renamed soon, so only use this as a temporary workaround.
Also, when "skipping" the API proxy, some things may not work as expected (for example, paths
used in bind-mounts are not rewritten, network settings may not be adapted for how Docker Desktop is setup)

☝️ ☝️ ☝️ ☝️ IOW: the recommended socket to use will be /var/run/docker.sock, which is the official API to connect (for reasons above).

reynoldsdj · 2020-11-18T14:54:59Z

I think this may be the same issue as docker/for-mac#5025 / docker/for-mac#5058 / docker/for-mac#5059. More details can be found in this comment: docker/for-mac#5059 (comment)

Yes, I believe that it the exact cause of this issue. docker/for-mac#5025

thaJeztah · 2020-11-18T17:49:02Z

looks like the issue should be fixed in the latest edge release that was just published docker/for-mac#5025 (comment)

kilrau · 2020-11-20T13:11:54Z

Latest edge doesn't fix this for me. Can someone confirm?

reynoldsdj · 2020-11-20T13:45:32Z

I concur. I just upgraded, and I am seeing the same error.

reynoldsdj · 2020-11-20T13:56:31Z

Error from trying to find a non-existent container (test suite log):

2020-11-20 08:42:56,620 - tests.fixtures.db - WARNING - Received ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) due to docker bug

thaJeztah · 2020-11-20T14:24:12Z

Can confirm that I still see the problem;

Incorrect response with the proxied socket;

$ curl -v --unix-socket /var/run/docker.sock http://localhost/containers/nosuchcontainer/json

*   Trying /var/run/docker.sock:0...
* Connected to localhost (docker.sock) port 80 (#0)
> GET /containers/nosuchcontainer/json HTTP/1.1
> Host: localhost
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Api-Version: 1.41
< Content-Type: application/json
< Date: Fri, 20 Nov 2020 14:23:21 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/20.10.0-rc1 (linux)
< Transfer-Encoding: chunked
<
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining

Works correct with the 'raw' socket:

$ curl -v --unix-socket ~/Library/Containers/com.docker.docker/Data/docker.raw.sock http://localhost/containers/nosuchcontainer/json

*   Trying /Users/sebastiaan/Library/Containers/com.dock:0...
* Connected to localhost (/Users/sebastiaan/Library/Containers/com.dock) port 80 (#0)
> GET /containers/nosuchcontainer/json HTTP/1.1
> Host: localhost
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Api-Version: 1.41
< Content-Type: application/json
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/20.10.0-rc1 (linux)
< Date: Fri, 20 Nov 2020 14:22:40 GMT
< Content-Length: 49
<
{"message":"No such container: nosuchcontainer"}
* Connection #0 to host localhost left intact

This works around docker/for-mac#5025 and docker/docker-py#2696

#22250) This works around docker/for-mac#5025 and docker/docker-py#2696

thaJeztah · 2020-12-01T16:22:50Z

This should be fixed in docker desktop 2.5.2 and up;

curl -v --unix-socket /var/run/docker.sock http://localhost/containers/nosuchcontainer/json

*   Trying /var/run/docker.sock:0...
* Connected to localhost (docker.sock) port 80 (#0)
> GET /containers/nosuchcontainer/json HTTP/1.1
> Host: localhost
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Api-Version: 1.41
< Content-Type: application/json
< Date: Tue, 01 Dec 2020 16:22:01 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/20.10.0-rc1 (linux)
< Transfer-Encoding: chunked
<
{"message":"No such container: nosuchcontainer"}
* Connection #0 to host localhost left intact

DmytroSokhach · 2021-02-04T17:32:22Z

@thaJeztah I'm trying to migrate from 2.3.2 to higher, we are uing docker stack like:

volumes:
   /var/run/docker.sock.raw:/var/run/docker.sock

Is there a way not to add ".raw", cause it is not backward compatible and will fail for those who stay on 2.3.2 (we have way to many non programmer users who will not know what to do with this)

milas · 2022-07-28T22:20:12Z

This is fixed in current versions of Docker Desktop.

Initially created in #22250 Both original issues are already solved for over 2.5 years: - docker/for-mac#5025 - docker/docker-py#2696 This will also allow anyone to use Orbstack without any modifications.

mconcas mentioned this issue Nov 4, 2020

Workaround for docker-py bug with macOS alidock/alidock#162

Merged

yannickperrenet mentioned this issue Nov 9, 2020

Docker networking issues for Mac orchest/orchest#56

Closed

johanhenriksson added a commit to backtick-se/cowait that referenced this issue Nov 18, 2020

implement workaround for docker for mac problem

6a727a7

fixes docker/docker-py#2696 by catching exceptions.

johanhenriksson mentioned this issue Nov 18, 2020

Workaround for Docker API response problem in Docker for Mac backtick-se/cowait#200

Merged

johanhenriksson added a commit to backtick-se/cowait that referenced this issue Nov 18, 2020

implement workaround for docker for mac problem

1bb2807

fixes docker/docker-py#2696 by catching exceptions.

thaJeztah mentioned this issue Nov 18, 2020

Docker-compose log streaming crash when containers stops or restarts docker/compose#7932

Closed

reliveyy mentioned this issue Nov 19, 2020

xud.sh not working on macOS ExchangeUnion/xud-docker#789

Open

mitsuhiko added a commit to getsentry/sentry that referenced this issue Nov 24, 2020

hack(devservices): A hacky workaround to make docker on mac work again

2daee93

This works around docker/for-mac#5025 and docker/docker-py#2696

mitsuhiko mentioned this issue Nov 24, 2020

hack(devservices): A hacky workaround to make docker on mac work again getsentry/sentry#22250

Merged

mitsuhiko added a commit to getsentry/sentry that referenced this issue Nov 24, 2020

hack(devservices): A hacky workaround to make docker on mac work again (

47f90c4

#22250) This works around docker/for-mac#5025 and docker/docker-py#2696

glicht mentioned this issue Nov 26, 2020

fail on unexpected exception + handle mac os docker demisto/demisto-sdk#904

Merged

5 tasks

milas closed this as completed Jul 28, 2022

RafaelFigueiredo mentioned this issue Jan 15, 2023

prefect agent Could not connect to docker PrefectHQ/prefect#7011

Closed

4 tasks

kamilogorek mentioned this issue Jul 27, 2023

ref: Remove workaround for docker sock in devservices.py getsentry/sentry#53688

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error while using docker for mac client 2.5.0.0 #2696

error while using docker for mac client 2.5.0.0 #2696

mattmoskwa commented Nov 2, 2020

mconcas commented Nov 3, 2020 •

edited

dkleissa commented Nov 3, 2020

dkleissa commented Nov 3, 2020

reynoldsdj commented Nov 4, 2020 •

edited

tophers42 commented Nov 5, 2020

kevinkwoncareem commented Nov 6, 2020

mconcas commented Nov 9, 2020

avirshup commented Nov 10, 2020

yannickperrenet commented Nov 10, 2020 •

edited

reynoldsdj commented Nov 10, 2020

jukkei commented Nov 10, 2020

ch3ck commented Nov 12, 2020

mconcas commented Nov 13, 2020 •

edited

thaJeztah commented Nov 18, 2020

thaJeztah commented Nov 18, 2020

thaJeztah commented Nov 18, 2020

reynoldsdj commented Nov 18, 2020

thaJeztah commented Nov 18, 2020

kilrau commented Nov 20, 2020

reynoldsdj commented Nov 20, 2020

reynoldsdj commented Nov 20, 2020

thaJeztah commented Nov 20, 2020

thaJeztah commented Dec 1, 2020

DmytroSokhach commented Feb 4, 2021

milas commented Jul 28, 2022

error while using docker for mac client 2.5.0.0 #2696

error while using docker for mac client 2.5.0.0 #2696

Comments

mattmoskwa commented Nov 2, 2020

mconcas commented Nov 3, 2020 • edited

dkleissa commented Nov 3, 2020

dkleissa commented Nov 3, 2020

reynoldsdj commented Nov 4, 2020 • edited

tophers42 commented Nov 5, 2020

kevinkwoncareem commented Nov 6, 2020

mconcas commented Nov 9, 2020

avirshup commented Nov 10, 2020

yannickperrenet commented Nov 10, 2020 • edited

reynoldsdj commented Nov 10, 2020

jukkei commented Nov 10, 2020

ch3ck commented Nov 12, 2020

mconcas commented Nov 13, 2020 • edited

thaJeztah commented Nov 18, 2020

thaJeztah commented Nov 18, 2020

thaJeztah commented Nov 18, 2020

reynoldsdj commented Nov 18, 2020

thaJeztah commented Nov 18, 2020

kilrau commented Nov 20, 2020

reynoldsdj commented Nov 20, 2020

reynoldsdj commented Nov 20, 2020

thaJeztah commented Nov 20, 2020

thaJeztah commented Dec 1, 2020

DmytroSokhach commented Feb 4, 2021

milas commented Jul 28, 2022

mconcas commented Nov 3, 2020 •

edited

reynoldsdj commented Nov 4, 2020 •

edited

yannickperrenet commented Nov 10, 2020 •

edited

mconcas commented Nov 13, 2020 •

edited