Getting 404 (image not found) when trying /v1/images/<image name>/ancestry

[vmalloc]

This doesn't happen consistently, but sometimes docker pull decides to request this path and stumbles upon 404 (and fails as a result).

Could this be a docker issue? Is it related to docker-registry?

Thanks in advance

[samalba]

Related issue: moby/moby#2461

[rogaha]

Hi @vmalloc,

Could please certify that your local time is synced?

Thanks

[vmalloc]

We're using ntp. From what I can tell both machines are synced. 

—
Sent from Mailbox for iPhone

On Thu, Nov 28, 2013 at 2:06 AM, Roberto Gandolfo Hashioka
notifications@github.com wrote:

Hi @vmalloc,
Could please certify that your local time is synced?

Thanks

Reply to this email directly or view it on GitHub:
#130 (comment)

[mzsanford]

I noticed this recently and dug into it. My docker pul call to /v1/repositories/mycompany/myname/images was returning an entry with an id which did not exist in the registry (not in the images directory). Once in this state the only way I could find out of it was to manually edit the _index_images to remove the offending entry. After that everything was back to normal. I'm still unsure how the registry was able to get into that state but once it did no amount of rebuilding mycompany/myname:latest seemed to fix it.

[gangleri]

I'm getting the same error. I have started the registry with:

sudo docker run -p 5000:5000 -v /home/gangleri/docker:/config -e DOCKER_REGISTRY_CONFIG=/config/config.yml -e SETTINGS_FLAVOR=dev -v /home/gangleri/docker/images/:/opt/images registry

The config file looks like:

common:
    loglevel: info
dev:
    loglevel: debug
    storage: local
    storage_path: /opt/images

test:
    storage: local
    storage_path: /tmp/tmpdockertmp

I then pull an image from the index:

sudo docker pull ubuntu

And tag it as follows

sudo docker tag 9cd978db300e localhost:5000/myubuntu

When push is being performed I see the following on the server

172.17.42.1 - - [20/Mar/2014:17:06:24] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
2014-03-20 17:06:24,371 INFO: 172.17.42.1 - - [20/Mar/2014:17:06:24] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
2014-03-20 17:06:24,376 DEBUG: check_session: Session is empty
172.17.42.1 - - [20/Mar/2014:17:06:24] "PUT /v1/repositories/myubuntu/ HTTP/1.1" 200 2 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64"
2014-03-20 17:06:24,378 INFO: 172.17.42.1 - - [20/Mar/2014:17:06:24] "PUT /v1/repositories/myubuntu/ HTTP/1.1" 200 2 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64"
2014-03-20 17:06:24,382 DEBUG: check_session: Session is empty
172.17.42.1 - - [20/Mar/2014:17:06:24] "GET /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json HTTP/1.1" 200 483 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd6
4"
2014-03-20 17:06:24,385 INFO: 172.17.42.1 - - [20/Mar/2014:17:06:24] "GET /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json HTTP/1.1" 200 483 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0
-31-generic os/linux arch/amd64"
2014-03-20 17:06:24,388 DEBUG: check_session: Session is empty
2014-03-20 17:06:24,389 DEBUG: api_error: Image not found
172.17.42.1 - - [20/Mar/2014:17:06:24] "GET /v1/images/6170bb7b0ad1003a827e4dc5253ba49f6719599eac485db51eaafd507c13c311/json HTTP/1.1" 404 34 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64
"
2014-03-20 17:06:24,391 INFO: 172.17.42.1 - - [20/Mar/2014:17:06:24] "GET /v1/images/6170bb7b0ad1003a827e4dc5253ba49f6719599eac485db51eaafd507c13c311/json HTTP/1.1" 404 34 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-
31-generic os/linux arch/amd64"
2014-03-20 17:06:24,394 DEBUG: check_session: Session is empty
172.17.42.1 - - [20/Mar/2014:17:06:24] "PUT /v1/images/6170bb7b0ad1003a827e4dc5253ba49f6719599eac485db51eaafd507c13c311/json HTTP/1.1" 200 4 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64"
2014-03-20 17:06:24,398 INFO: 172.17.42.1 - - [20/Mar/2014:17:06:24] "PUT /v1/images/6170bb7b0ad1003a827e4dc5253ba49f6719599eac485db51eaafd507c13c311/json HTTP/1.1" 200 4 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-3
1-generic os/linux arch/amd64"
2014-03-20 17:06:24,423 DEBUG: check_session: Session is empty
2014-03-20 17:06:24,424 DEBUG: checksums.compute_tarsum: return tarsum+sha256:3baf0dc0e2713fe1b705feee05d7221a0e4859c901f0f33c393a63adb5c208a1
172.17.42.1 - - [20/Mar/2014:17:06:24] "PUT /v1/images/6170bb7b0ad1003a827e4dc5253ba49f6719599eac485db51eaafd507c13c311/layer HTTP/1.1" 200 4 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64
"
2014-03-20 17:06:24,426 INFO: 172.17.42.1 - - [20/Mar/2014:17:06:24] "PUT /v1/images/6170bb7b0ad1003a827e4dc5253ba49f6719599eac485db51eaafd507c13c311/layer HTTP/1.1" 200 4 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-
31-generic os/linux arch/amd64"
2014-03-20 17:06:24,428 DEBUG: check_session: Session is empty
2014-03-20 17:06:24,429 DEBUG: api_error: Checksum not found in Cookie
172.17.42.1 - - [20/Mar/2014:17:06:24] "PUT /v1/images/6170bb7b0ad1003a827e4dc5253ba49f6719599eac485db51eaafd507c13c311/checksum HTTP/1.1" 400 47 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/a
md64"
2014-03-20 17:06:24,431 INFO: 172.17.42.1 - - [20/Mar/2014:17:06:24] "PUT /v1/images/6170bb7b0ad1003a827e4dc5253ba49f6719599eac485db51eaafd507c13c311/checksum HTTP/1.1" 400 47 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.
8.0-31-generic os/linux arch/amd64"

On the client I see

sudo docker push localhost:5000/myubuntu                                                                                                                           
The push refers to a repository [localhost:5000/myubuntu] (len: 1)
Sending image list
Pushing repository localhost:5000/myubuntu (1 tags)
Image 511136ea3c5a already pushed, skipping
6170bb7b0ad1: Pushing [==================================================>] 3.072 kB/3.072 kB
2014/03/20 17:06:24 }

When I try and pull this image I get the following error:

sudo docker pull localhost:5000/myubuntu                                                                                                                           
Pulling repository localhost:5000/myubuntu
9cd978db300e: Error pulling image (latest) from localhost:5000/myubuntu, Server error: 404 trying to fetch remote history for 9cd978db300e27386baa9dd791bf6dc818f13e52235b26e95703361ec3c94dc6 
2014/03/20 17:10:23 Could not find repository on any of the indexed registries.

And in the registry I see

^[[A172.17.42.1 - - [20/Mar/2014:17:10:23] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
2014-03-20 17:10:23,441 INFO: 172.17.42.1 - - [20/Mar/2014:17:10:23] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
2014-03-20 17:10:23,444 DEBUG: check_session: Session is empty
172.17.42.1 - - [20/Mar/2014:17:10:23] "GET /v1/repositories/myubuntu/images HTTP/1.1" 200 238 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64"
2014-03-20 17:10:23,446 INFO: 172.17.42.1 - - [20/Mar/2014:17:10:23] "GET /v1/repositories/myubuntu/images HTTP/1.1" 200 238 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64"
2014-03-20 17:10:23,449 DEBUG: check_session: Session is empty
2014-03-20 17:10:23,449 DEBUG: [get_tags] namespace=library; repository=myubuntu
172.17.42.1 - - [20/Mar/2014:17:10:23] "GET /v1/repositories/library/myubuntu/tags HTTP/1.1" 200 2 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64"
2014-03-20 17:10:23,451 INFO: 172.17.42.1 - - [20/Mar/2014:17:10:23] "GET /v1/repositories/library/myubuntu/tags HTTP/1.1" 200 2 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64"
2014-03-20 17:10:23,454 DEBUG: check_session: Session is empty
2014-03-20 17:10:23,454 DEBUG: api_error: Image not found
172.17.42.1 - - [20/Mar/2014:17:10:23] "GET /v1/images/9cd978db300e27386baa9dd791bf6dc818f13e52235b26e95703361ec3c94dc6/ancestry HTTP/1.1" 404 34 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64"
2014-03-20 17:10:23,456 INFO: 172.17.42.1 - - [20/Mar/2014:17:10:23] "GET /v1/images/9cd978db300e27386baa9dd791bf6dc818f13e52235b26e95703361ec3c94dc6/ancestry HTTP/1.1" 404 34 "-" "docker/0.9.0 go/go1.2.1 git-commit/2b3fdf2 kernel/3.8.0-31-generic os/linux arch/amd64"

When I browse to the images directory of the registry I see the following entries

1c7f181e78b90d347996d754ffa38c4c6b395e7cf0388bffffbda00365b45077
27cf784147099545
511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158
58394af373423902a1b97f209a31e3777932d9321ef10e64feaaa7b4df609cf9
6170bb7b0ad1003a827e4dc5253ba49f6719599eac485db51eaafd507c13c311
8abc22fbb04266308ff408ca61cb8f6f4244a59308f7efc64e54b08b496c58db
9f676bd305a43a931a8d98b13e5840ffbebcd908370765373315926024c7c35e
b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc

docker version gives the following output

Client version: 0.9.0
Go version (client): go1.2.1
Git commit (client): 2b3fdf2
Server version: 0.9.0
Git commit (server): 2b3fdf2
Go version (server): go1.2.1
Last stable version: 0.9.0

[kscherer]

I am having a similar problem using the registry container on the Docker index. I was able to work around it by updating the git repo in the container to the latest.

docker run -e DOCKER_REGISTRY_CONFIG=/registry/config.yml -p 5000:5000 -v /opt/registry:/registry -m 0 registry /bin/sh -c 'cd /docker-registry && git remote add origin https://github.com/dotcloud/docker-registry.git && git pull origin master && ./setup-configs.sh && exec ./run.sh'

The documentation suggests that the "preferred" way of running a private registry is to use the registry container in the docker index, but the fact that:

• it is out of date
• does not use nginx
• does not rotate the logs

suggests that the registry container still needs some work.

[samalba]

cc @shin-

[pmyjavec]

Yup, same for me, the official image seems to be out of date. As soon as I run the latest from the repo everything seems to work perfectly again.

Happy to help out in anyway I can, getting this thing running against a CI server like Travis or something.