Docker::Error::ServerError: error parsing HTTP 400 response body: unexpected end of JSON input: ""

[matti]

deployed 2 instances of the stack successfully, then on the third stack install + deploy:

TIME                      TYPE                      MESSAGE
2017-07-24T12:37:58.875Z  service:create_instance   pulling image registry.example.com/mystack/daemon:latest for mystack/daemon-1 (node-1)
2017-07-24T12:38:03.166Z  service:create_instance   unexpected error while creating mystack/daemon-1: error parsing HTTP 400 response body: unexpected end of JSON input: ""
 (node-1)
Deployment of service mygrid/mystack/daemon failed:
  - halting deploy of mygrid/mystack/daemon, one or more instances failed
  - GridServiceInstanceDeployer::ServiceError: Docker::Error::ServerError: error parsing HTTP 400 response body: unexpected end of JSON input: "" (on node node-1)

$ cat /etc/lsb-release
DISTRIB_ID="Container Linux by CoreOS"
DISTRIB_RELEASE=1409.7.0
DISTRIB_CODENAME="Ladybug"
DISTRIB_DESCRIPTION="Container Linux by CoreOS 1409.7.0 (Ladybug)"

$ docker -v
Docker version 1.12.6, build a82d35e

[SpComb]

Anything interesting in journalctl -u docker.service from the same time?

Sounds like a Docker daemon error.

[matti]

okay, resolved: repository didn't exist:

Jul 24 13:33:08 node-1 dockerd[992]: time="2017-07-24T13:33:08.766932539Z" level=error msg="Error trying v2 registry: error parsing HTTP 400 response body: unexpected end of JSON input: \"\""
Jul 24 13:33:08 node-1 dockerd[992]: time="2017-07-24T13:33:08.767477407Z" level=error msg="Attempting next endpoint for pull after error: error parsing HTTP 400 response body: unexpected end of JSON input: \"\""
Jul 24 13:33:08 node-1 dockerd[992]: time="2017-07-24T13:33:08.767703295Z" level=error msg="Handler for POST /v1.22/images/create returned error: error parsing HTTP 400 response body: unexpected end of JSON input: \"\""
Jul 24 13:33:09 node-1 dockerd[992]: time="2017-07-24T13:33:09.299070832Z" level=error msg="Error trying v2 registry: error parsing HTTP 400 response body: unexpected end of JSON input: \"\""
Jul 24 13:33:09 node-1 dockerd[992]: time="2017-07-24T13:33:09.299614815Z" level=error msg="Attempting next endpoint for pull after error: error parsing HTTP 400 response body: unexpected end of JSON input: \"\""
Jul 24 13:33:09 node-1 dockerd[992]: time="2017-07-24T13:33:09.299868935Z" level=error msg="Handler for POST /v1.22/images/create returned error: error parsing HTTP 400 response body: unexpected end of JSON input: \"\""
Jul 24 13:33:09 node-1 dockerd[992]: time="2017-07-24T13:33:09.804902465Z" level=error msg="Error trying v2 registry: error parsing HTTP 400 response body: unexpected end of JSON input: \"\""
Jul 24 13:33:09 node-1 dockerd[992]: time="2017-07-24T13:33:09.805444988Z" level=error msg="Attempting next endpoint for pull after error: error parsing HTTP 400 response body: unexpected end of JSON input: \"\""
Jul 24 13:33:09 node-1 dockerd[992]: time="2017-07-24T13:33:09.805663823Z" level=error msg="Handler for POST /v1.22/images/create returned error: error parsing HTTP 400 response body: unexpected end of JSON input: \"\""

Should this issue be closed or should the error handling be improved?

[SpComb]

Based on those logs, it looks like the Docker::Image.create may have failed with raise Docker::Error::ServerError, 'error parsing HTTP 400 response body: unexpected end of JSON input: ""'. It looks like this isn't the agent docker API client getting an unexpected EOF from the local Docker daemon, but the Docker daemon getting a HTTP 400 response + unexpected EOF from the external image registry?

IOW: whatever registry you're pulling from seems to be broken?

The agent could perhaps give some better error with more context, like Docker server error while pulling image foo.io/bar: error parsing HTTP 400 response body: unexpected end of JSON input: ""'

[matti]

yep. Heroku's container registry apparently returns "" if image not found

[andreatulimiero]

IMHO the error should be clarified since a response code "400" is truly misleading

[blairg]

I had a similar issue with the 400 response. This is what helped us https://hackerlite.xyz/heroku-docker-deployments/