image.push doesn't fail if docker daemon fails to push image to registry

[neurogenesis]

We're having a problem pushing images to a 3rd-party container registry (quay.io), which seems to stem from a mismatch in API versions.

We can push an image from the docker command-line, which uses API v1.19 (POST /v1.19/images/...), whereas image.push via docker daemon results in failed push attempts (which appears to use API v1.15, POST /v1.15/images...).

Problems with docker daemon and quay.io aside, we would expect image.push to fail if docker cannot push the image to the registry. It would appear this is an asynchronous call at the moment, without a way to inspect the result / success of the call.

https://github.com/swipely/docker-api/blob/master/lib/docker/image.rb#L24-L38

[tlunter]

@neurogenesis Can you try using a block with image.push?

image = Docker::Image.all.first
image.push do |chunk|
  puts chunk
end

This'll show you what docker is outputting to us. If they're returning a 200 but failing, this is definitely an API problem.

[neurogenesis]

In this specific case, quay.io is returning a 500...

{"status":"The push refers to a repository [quay.io/REPO/IMAGE] (len: 1)"}
{"status":"Sending image list"}
{"errorDetail":{"code":500,"message":"Error: Status 500 trying to push repository REPO/IMAGE: \"\u003c!DOCTYPE HTML PUBLIC \\\"-//W3C//DTD HTML 3.2 Final//EN\\\"\u003e\\n\u003ctitle\u003e500 Internal Server Error\u003c/title\u003e\\n\u003ch1\u003eInternal Server Error\u003c/h1\u003e\\n\u003cp\u003eThe server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.\u003c/p\u003e\\n\""},"error":"Error: Status 500 trying to push repository REPO/IMAGE: \"\u003c!DOCTYPE HTML PUBLIC \\\"-//W3C//DTD HTML 3.2 Final//EN\\\"\u003e\\n\u003ctitle\u003e500 Internal Server Error\u003c/title\u003e\\n\u003ch1\u003eInternal Server Error\u003c/h1\u003e\\n\u003cp\u003eThe server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.\u003c/p\u003e\\n\""}

However, regardless of the failure reason, I would have expected image.push to return/raise an error if the action of pushing the image failed. The workaround is to inspect the returned chunked messages (as you pointed out above).

The odd thing here is that issuing the following from the commandline will work:

sudo docker push [repo]:[tag]

In our case, the request sent via docker CLI appears to using a more recent API version when issuing the request to quay.io than via docker-api. By the time quay.io gets the request from docker deamon, they interpret it as a bad request. This is a different issue altogether though. Still trying to capture what docker daemon is sending (or omitting) in the two different requests to quay.io.

[neurogenesis]

To the original point however, it doesn't feel like #push is as deterministic as it should be. I would expect push to raise an error if the response was anything other than 200. As an API call, I would also expect to have access to Docker APIs response code in addition to the chunked messages.

The underlying call seems to do this here:

https://docs.docker.com/reference/api/docker_remote_api_v1.16/

    HTTP/1.1 200 OK
    Content-Type: application/json

    {"status": "Pushing..."}
    {"status": "Pushing", "progress": "1/? (n/a)", "progressDetail": {"current": 1}}}
    {"error": "Invalid..."}
    ...

[tlunter]

@neurogenesis I'd like to clarify first: it's not that we're dropping the response on the ground. The docker daemon is not erroring when it cannot push to a registry like quay. Docker is returning to us a 200, which is just as alarming to me as it is to you. It seems their API is either broken or designed without the intention of letting the user know something isn't working.

If you run your code with EXCON_DEBUG=true you'll quickly see that we're actually getting a 200 from the daemon, and that is what is failing. It looks like the best way for you to understand if the docker daemon is working or not is to parse the chunked response. From your JSON above, it looks like quay is returning a 500 response code to the docker daemon and the daemon is forwarding this to you.

If we were to do this ourselves, we would do just that. You can see here: https://github.com/swipely/docker-api/blob/master/lib/docker/connection.rb#L88 that we're expecting only valid response codes, and here: https://github.com/swipely/docker-api/blob/master/lib/docker/connection.rb#L41-L52 that we've seen other response codes in the past.

We've struggled with parsing dockers chunked response errors since they do not always come in the right order and have changed messaging many times. https://github.com/swipely/docker-api/blob/master/lib/docker/image.rb#L116-L117 You can find the history of these two lines going back quite a few revisions, and that is still broken: #315.

It seems something is broken in the daemon or in quay if it's returning a 500. Docker did accept our request, so the problem probably lies farther downstream. Have you authenticated properly? Maybe quay does not have the best error message. What form of authentication are you using? Perhaps docker doesn't realize something is broken as well.

[neurogenesis]

@tlunter , thanks for the additional context. I definitely understand the challenge you're facing with the Docker API now. From their own API docs, it looks like they should be returning non-2xx response codes when there are errors:

http://docs.docker.com/engine/reference/api/docker_remote_api_v1.16/#push-an-image-on-the-registry

The chunked responses, while handy for large output like docker logs, seems inferior to a single JSON object (in terms of both performance and deterministic parsing & control of the Docker Daemon).

Closing this issue then.

[anklos]

@tlunter I doubt docker daemon always return 200, since if I do system("docker push #{some_wrong_directory}"), it returns false in the console.

my docker version is 1.8

[tlunter]

@anklos Even though the docker CLI is returning a non-zero exit code, doesn't mean that their API is returning a non-200 status. Nothing says that a 200 status code from the remote API translates to success with docker push x. The docker client actually does quite a bit under the hood, and that's likely what is going on here. If we were to look at the client code, I'm sure we'd find that they are parsing the output of the API request and error if they see an error key in the JSON.

I didn't say that every API request will result in a 200, just this one that the original issue was made regarding. If you run with EXCON_DEBUG=true, you'll likely see similar output to @neurogenesis and that can help pin-point your error and where to go next. I also recommend you run the docker daemon with the debug flag to help you trace down what is failing since in @neurogenesis's case, the error was being pushed over the API from the daemon.

[anklos]

@tduffield thanks!

As you suggested using a block with push, can I do something like this?

tag = docker_image_full_name.split(':')[1]
flag = false
Docker::Image.get(docker_image_full_name).push do |chunk|
  if chunk && chunk.include?(tag)
    flag = true
  end
end

flag

Because I can see always something like "{\"status\":\"26_2b06eb4_UAT_240_4b703ad: digest: sha256:386953d6a96c9f79285e70efafe82f108afc44e1524b12b2f8d5bf3501964cb0 size: 19672\"}\r\n" when the push succeeds.

[tlunter]

Yes, you can either do something like that, or:

flag = true

Docker::Image.get(docker_image_full_name).push do |chunk|
  json = JSON.parse(chunk)
  flag = false if json['error']
end

flag

This looks for the error key that should be present only if there is an error.

[anklos]

@tlunter cool, thanks!

[universam1]

For me it was necessary to split the lines:

  Docker::Image.get(docker_image_full_name).push do |chunk|
    chunk.each_line do |l|
      puts l
      json = JSON.parse(l)
      errorflag = true if json['error']
    end
  end