streaming from build() is broken in 0.3.0

[rosenhouse]

Some client code that was working fine w/ docker-py 0.2.3 now breaks on 0.3.0

A minimal test case looks like:

ubuntu@ip-10-77-1-34:/tmp/foo$ cat Dockerfile
FROM stackbrew/ubuntu:13.04
RUN echo "hello"
CMD /bin/bash

ubuntu@ip-10-77-1-34:/tmp/foo$ cat test.py
import docker

doc = docker.Client()
gen = doc.build(path='.', stream=True)
for line in gen:
    print line

Executing test.py under 0.2.3 builds the image fine. Under 0.3.0, I get

ubuntu@ip-10-77-1-34:/tmp/foo$ python test.py
{"stream":" ---\u003e Using cache\n"}

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    for line in gen:
  File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 239, in _stream_helper
    size = int(socket.readline(), 16)
ValueError: invalid literal for int() with base 16: ''

This is Ubuntu 12.04 w/ Docker 0.9.0

[shin-]

Thanks for the report! Which Python version are you using?

[rosenhouse]

ah, I figured I'd forgotten something relevant: 2.7.3. Cheers.

[ereyes01]

Same problem in Python 2.7.5.

Workaround for me was to force version='1.6' in the Client constructor.

[ereyes01]

Sorry, I looked more closely, and my problem is close, but not the same as rosenhouse's problem. I get this problem without specifying stream=True in 0.3.0, but it does involve the same _stream_helper() code.

Sorry if I'm hijacking this for something that should be a different issue, but I'll post my findings.

Here is what I see:

In [3]: c = docker.Client(base_url='http://localhost:4243')

In [4]: r = c.build(path='/tmp/tmpdEEGN8', tag='eddy-test')

In [5]: r
Out[5]: <generator object _stream_helper at 0x2263960>

...

In [7]: r.next()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-8d25240d5a85> in <module>()
----> 1 r.next()

/home/ereyes/work/new-env/local/lib/python2.7/site-packages/docker/client.pyc in _stream_helper(self, response)
    233     def _stream_helper(self, response):
    234         """Generator for data coming from a chunked-encoded HTTP response."""
--> 235         socket_fp = self._get_raw_response_socket(response)
    236         socket_fp.setblocking(1)
    237         socket = socket_fp.makefile()

/home/ereyes/work/new-env/local/lib/python2.7/site-packages/docker/client.pyc in _get_raw_response_socket(self, response)
    229     def _get_raw_response_socket(self, response):
    230         self._raise_for_status(response)
--> 231         return response.raw._fp.fp._sock
    232 
    233     def _stream_helper(self, response):

AttributeError: 'NoneType' object has no attribute '_sock'

The image does get built, but my code was relying on the return value of build() to get at the image ID.

It looks like, by default as of "version 1.8", build will always assume you want to stream the output back:

        if stream or utils.compare_version('1.8', self._version) >= 0:
            return self._stream_helper(response)
        else:
            output = self._result(response)

This is why the workaround for me was to force version='1.6' in the client. This worked for me because I didn't want to stream at all. I would imagine that if you really want to do stream=True, you may still be stuck if you force the version to something older like 1.6.

[mpetazzoni]

The reason for this is that from API version 1.8 onwards the /build endpoint IIRC always returns a stream.

[shin-]

Yes, I think these are 2 different issues. Absence of image id for version >= 1.8 is intended, errors when trying to read from the stream are not.

[ibuildthecloud]

I'm getting this issue too but on pull(stream=True). Ubuntu 13.10, Python 2.6.9 (yeah, i know that's a little odd), Docker 0.9.0, Docker-py 0.3.0.

    def _stream_helper(self, response):
        """Generator for data coming from a chunked-encoded HTTP response."""
        socket_fp = self._get_raw_response_socket(response)
        socket_fp.setblocking(1)
        socket = socket_fp.makefile()
        while True:
>           size = int(socket.readline(), 16)
E           ValueError: invalid literal for int() with base 16: ''

Is the root cause of this know? I'm going to dig into this tonight, as it is breaking some stuff for me. Let me know if the correct fix is already known.

[ibuildthecloud]

So basically what I'm seeing is that between 0.8.x and 0.9.0, due to moby/moby#4276, docker now sends empty newline chunks in streamed response.

So before the response would look line

42\r\n
{......}\r\n
42\r\n
{......}\r\n

Now it looks like

42\r\n
{......}\r\n
\r\n
42\r\n
{......}\r\n
\r\n

So I think just a simple check to ignore the newlines will probably fix it. I'll try it out.

[ibuildthecloud]

Ah, I see what's happening now. So _stream_helper reads the chunk size, but then doesn't actually use the size to read the data

    def _stream_helper(self, response):
        """Generator for data coming from a chunked-encoded HTTP response."""
        socket_fp = self._get_raw_response_socket(response)
        socket_fp.setblocking(1)
        socket = socket_fp.makefile()
        while True:
            size = int(socket.readline(), 16)
            if size <= 0:
                break
            data = socket.readline()
            if not data:
                break
            yield data

So if the data has a newline in it (which in 0.8.x never happened), then socket.readline() will read a partial data chunk. Then when the loop starts again, where you are expecting to read an int, you are really still reading the chunk.

I'm going to experiment with switching to using Request.iter_lines() so that docker-py doesn't have to be so low level and read chunks