Cannot access docker daemon using docker-py

[kurtwheeler]

I am trying to rewrite my deployment script in python instead of bash. Part of that script requires calling docker. I'd like to use the docker-py client to build and push the image instead of having to use subprocess to call the docker CLI. However I cannot get docker-py to communicate with the Docker daemon. All of the examples start with a call to docker.from_env() and then assumes the client just works from there, however this seems to be the hardest part of using docker-py.

Whether I call:

client = docker.DockerClient(base_url="unix://var/run/docker.sock", version="auto")
client.images.build(
    path="../",
    dockerfile="Dockerfile.prod",
    tag=image_name,
    buildargs={"SYSTEM_VERSION": args.system_version, "HTTP_PORT": HTTP_PORT},
)

or

client = docker.from_env(version="auto")
client.images.build(
    path="../",
    dockerfile="Dockerfile.prod",
    tag=image_name,
    buildargs={"SYSTEM_VERSION": args.system_version, "HTTP_PORT": HTTP_PORT},
)

the result is the same:

$ python3 deploy.py -d wkurt -e dev -u kurt -v v0.0.0
Traceback (most recent call last):
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "deploy.py", line 71, in <module>
    buildargs={"SYSTEM_VERSION": args.system_version, "HTTP_PORT": HTTP_PORT},
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/docker/models/images.py", line 279, in build
    resp = self.client.api.build(**kwargs)
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/docker/api/build.py", line 269, in build
    timeout=timeout,
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/docker/api/client.py", line 226, in _post
    return self.post(url, **self._set_request_timeout(kwargs))
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/kurt/Development/resources-portal/infrastructure/rpenv/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

However I don't think there's any issue with my Docker configuration or the way I'm invoking my python script because if I call docker through subprocess it works:

os.chdir("../")

system_version_build_arg = "SYSTEM_VERSION={}".format(args.system_version)
http_port_build_arg = "HTTP_PORT={}".format(HTTP_PORT)
completed_command = subprocess.run(
    [
        "docker",
        "build",
        "--tag",
        image_name,
        "--build-arg",
        system_version_build_arg,
        "--build-arg",
        http_port_build_arg,
        "-f",
        "Dockerfile.prod",
        ".",
    ],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

command_output = (
    completed_command.stdout.decode().strip() + completed_command.stderr.decode().strip()
)
print(command_output)

outputs:

$ python3 deploy.py -d wkurt -e dev -u kurt -v v0.0.0
Sending build context to Docker daemon  155.1MB
Step 1/18 : FROM python:3.8
 ---> f88b2f81f83a
Step 2/18 : ENV PYTHONUNBUFFERED 1
 ---> Using cache
 ---> 034aea58b08f
Step 3/18 : RUN groupadd user && useradd --create-home --home-dir /home/user -g user user
 ---> Using cache
 ---> 1ccd20fff4c5
Step 4/18 : WORKDIR /home/user
 ---> Using cache
 ---> 6f5a157ef2e8
Step 5/18 : COPY ./requirements.txt requirements.txt
 ---> Using cache
 ---> 3b348c8dad3a
Step 6/18 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> c5c1a09a6e1a
Step 7/18 : COPY . .
 ---> 466d2f86df71
Step 8/18 : RUN chmod +x /home/user/collect_and_run_uwsgi.sh
 ---> Running in 5721e33787aa
Removing intermediate container 5721e33787aa
 ---> 00e7c17a2bc7
Step 9/18 : RUN mkdir -p /tmp/www/static
 ---> Running in 404e3fcbd252
Removing intermediate container 404e3fcbd252
 ---> 8b8b709b5127
Step 10/18 : RUN chown user /tmp/www/static
 ---> Running in 99148e62595d
Removing intermediate container 99148e62595d
 ---> 3dcbbc754c4e
Step 11/18 : ARG SYSTEM_VERSION
 ---> Running in 3a30f5495169
Removing intermediate container 3a30f5495169
 ---> 7c8b7f41b9dd
Step 12/18 : ARG HTTP_PORT
 ---> Running in 40a598ba942e
Removing intermediate container 40a598ba942e
 ---> c22dd2e10986
Step 13/18 : ENV SYSTEM_VERSION $SYSTEM_VERSION
 ---> Running in 382021f3fca4
Removing intermediate container 382021f3fca4
 ---> 846ebf3919a9
Step 14/18 : ENV HTTP_PORT ":$HTTP_PORT"
 ---> Running in c4cf64353108
Removing intermediate container c4cf64353108
 ---> 205dc12c4b7c
Step 15/18 : USER user
 ---> Running in 41e6f9ecee45
Removing intermediate container 41e6f9ecee45
 ---> 615f57cc819c
Step 16/18 : VOLUME /tmp/www/static
 ---> Running in 4fa5712b2cb3
Removing intermediate container 4fa5712b2cb3
 ---> e62ec3486a8b
Step 17/18 : EXPOSE $HTTP_PORT
 ---> Running in 64f2e260f2d6
Removing intermediate container 64f2e260f2d6
 ---> 7ca54271857e
Step 18/18 : CMD ./collect_and_run_uwsgi.sh
 ---> Running in 47dd529fd0a2
Removing intermediate container 47dd529fd0a2
 ---> ff2cf0b4346b
Successfully built ff2cf0b4346b
Successfully tagged wkurt/resources_portal_api:v0.0.0

Does anyone have any ideas why docker works when accessed via the CLI (even if from within python) but docker-py doesn't? Thanks!

[ryanjdillon]

I am having a similar problem on CircleCI, which I believe is because the docker engine for the docker executor type is on an external machine (per here).

If you installed things to a non-standard location, I'm guessing you'd get the same error. So perhaps try specifying a different base url for the Client.

>>> import docker
>>> client = docker.DockerClient(base_url='unix://<docker daemon socket location>')

[kurtwheeler]

As far as I can tell, my docker is using the standard docker socket: /var/run/docker.sock, which is what I specified in:

client = docker.DockerClient(base_url="unix://var/run/docker.sock", version="auto")

Is there a command or a log somewhere that will output the currently-being-used socket? It doesn't seem to be output by docker system info.

[thaJeztah]

I'm not a python dev, but I noticed your socket location is one slash short; it's unix://<path> where path is an absolute path, so starts with a slash; so there should be three slashes after unix:;

 unix:///var/run/docker.sock

[kurtwheeler]

Sorry, it's been a bit since I've had a chance to poke at this. Adding an additional / to the socket path as per @thaJeztah's suggestion didn't change the output at all.

[kurtwheeler]

I will note that I've installed a fresh Ubuntu 18.04 since the last time I tried this, so it doesn't seem to be something in particular about my last system, unless I've repeated the same mistake on this one.

[mistafunk]

I had similar issues until I made sure all buildargs are strings... maybe that helps here as well?

[feliperuhland]

Hi All.

That's strange behavior.

To make everything clear, you CAN'T access through docker-py but CAN access thought shell/subprocess?

Could you share docker/docker-py versions?

Have a nice day :)