v1 registry - error pulling images with docker 1.8 / OEL 7.1

[vshiva]

I'm on docker 1.8 (docker-engine.x86_64 / 1.8.0-1.el7 )
I'm trying to pull images hosted on my internal corporate registry.

using
docker pull registry.mycorp.com/myfavimg
this results in
Could not reach any registry endpoint

However I'm able to pull from docker hub without any issue

$ docker info
Containers: 0
Images: 15
Storage Driver: devicemapper
Pool Name: docker-252:1-104464387-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 2.617 GB
Data Space Total: 107.4 GB
Data Space Available: 104.8 GB
Metadata Space Used: 2.404 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.145 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /scratch/docker/devicemapper/devicemapper/data
Metadata loop file: /scratch/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.8.13-98.el7uek.x86_64
Operating System: Oracle Linux Server 7.1
CPUs: 24
Total Memory: 141.4 GiB
ID: GFRH:Y3RH:66XS:XED5:N7KJ:V4RG:O4M6:INTA:DDIL:BNLL:DJ3N:ARCM

*$ docker version *
Client:
Version: 1.8.0
API version: 1.20
Go version: go1.4.2
Git commit: 0d03096
Built: Tue Aug 11 16:48:33 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.8.0
API version: 1.20
Go version: go1.4.2
Git commit: 0d03096
Built: Tue Aug 11 16:48:33 UTC 2015
OS/Arch: linux/amd64

[thaJeztah]

However I'm able to pull from docker hub without any issue

IIRC, Docker Hub is using both a V1 and V2 registry, where Docker 1.6(?) and above clients will automatically use the V2 endpoints.

A hybrid setup was described in the old documentation http://docs.docker.com/v1.6/registry/deploying/#configure-nginx-with-a-v1-and-v2-registry but I'm not sure if that setup is still supported.

@dmp42 should docker 1.8 still be able to talk to a 1.0 registry?

[dmp42]

@vshiva can you provide docker daemon logs (in debug mode)?

Thanks.

[vshiva]

$ uname -a
Linux slcac663.us.oracle.com 3.8.13-98.el7uek.x86_64 #2 SMP Mon Jul 27 10:45:45 PDT 2015 x86_64 x86_64 x86_64 GNU/Linux

command tried
docker pull docker.xcorp.com/idcs/x-xe-11g:latest

docker daemon logs (debug mode)

DEBU[0054] Calling POST /images/create
INFO[0054] POST /v1.20/images/create?fromImage=docker.xcorp.com%2Fidcs%2Fx-xe-11g%3Alatest
DEBU[0054] hostDir: /etc/docker/certs.d/docker.xcorp.com
DEBU[0054] Trying to pull docker.xcorp.com/idcs/x-xe-11g from https://docker.xcorp.com v2
DEBU[0054] Error getting v2 registry: endpoint does not support v2 API
DEBU[0054] Trying to pull docker.xcorp.com/idcs/x-xe-11g from https://docker.xcorp.com v1
DEBU[0054] hostDir: /etc/docker/certs.d/docker.xcorp.com
DEBU[0054] attempting v2 ping for registry endpoint https://docker.xcorp.com/v2/
DEBU[0054] attempting v1 ping for registry endpoint https://docker.xcorp.com/v1/
DEBU[0054] Error unmarshalling the _ping PingResult: json: cannot unmarshal bool into Go value of type registry.PingResult
DEBU[0054] PingResult.Version: ""
DEBU[0054] Registry standalone header: 'True'
DEBU[0054] PingResult.Standalone: true
DEBU[0054] Endpoint https://docker.xcorp.com/v1/ is eligible for private registry. Enabling decorator.
DEBU[0054] [registry] Calling GET https://docker.xcorp.com/v1/repositories/idcs/x-xe-11g/images
DEBU[0054] Retrieving the tag list
DEBU[0054] Got status code 405 from https://docker.xcorp.com/v1/repositories/idcs/x-xe-11g/tags/latest
ERRO[0054] unable to get remote tags: Could not reach any registry endpoint
DEBU[0054] Not continuing with error: Could not reach any registry endpoint

[puja108]

Seems to be the same endpoint problem like in #15558
We're experiencing the same issues with a self-hosted registry, everything works fine with pre-1.8 clients, though.

[dmp42]

@puja108 These are definitely different issues. 404 on #15558 which indicates a missing endpoint - 405 here.

@vshiva can you provide your registry logs?

Are you using any sort of proxy? nginx? apache?

[CWSpear]

❯❯❯ docker version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 19:47:52 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:49:29 UTC 2015
 OS/Arch:      linux/amd64

❯❯❯ docker info
Containers: 0
Images: 0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.1 (TCL 6.3); master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: docker
ID: WIHL:JSS5:WLZA:W6DM:TNJX:WJPF:BGXB:4LS7:HBHB:JNQQ:VO6D:JBSJ
Debug mode (server): true
File Descriptors: 10
Goroutines: 17
System Time: 2015-08-17T22:44:27.019608301Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=vmwarefusion

❯❯❯ uname -a
Darwin Dry-Bones.local 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64

Environment details: on a Mac, using docker-machine and vmware fusion. Updated everything today.

How reproducible: very

Steps to Reproduce:

1. docker pull "private-registry.com/elasticsearch"

Actual Results: Could not reach any registry endpoint

Expected Results: it downloads the image

Additional info: This works in 1.7.x. Our registry is using artifactory, and we do not support v2, I guess. I requested logs, to see if they'd help, but I am not holding my breath.

We think it's related to this issue, and that it's a v1/v2 issue, because in trying to get things to work, we did try this and saw that it was trying to hit a v2 URL (I couldn't figure out how to get docker pull to tell me the actual endpoints it was trying to hit, which is frustrating, since that's the error):

❯❯❯ export DOCKER_CONTENT_TRUST=1
❯❯❯ docker pull private-registry.com/elasticsearch:1.7.1
Error establishing connection to trust repository: Get https://private-registry.com/v2/: remote error: handshake failure
Get https://private-registry.com/v2/: remote error: handshake failure

I didn't necessarily expect that command to work, but I would have expected it to mention something about Get https://private-registry.com/v1/ (as opposed to v2) in the error message. Anyway... that's why we think it's an issue of docker 1.8.1 not playing nice with v1 registries... but admittedly, it may not have anything to do with anything! =)

I'm not using a firewall or anything, and images like coreos/etcd work fine.

Edit: I updated my docker version with the stuff from when I was using 1.8.1 on the client (originally pasted output from 1.7.1)

[CWSpear]

If I curl (with auth) https://private-registry.com/v1/repositories/library/elasticsearch/tags, I get these headers, if that helps you out any (I see questions about someone else's registry set up, and this is the best I can do without more access):

Request:

GET /v1/repositories/library/elasticsearch/tags HTTP/1.1
Host: private-registry.com
Authorization: redacted
User-Agent: curl/7.43.0
Accept: /

Response:

HTTP/1.1 200 OK
Server: nginx/1.6.3
Date: Mon, 17 Aug 2015 22:36:24 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
X-Artifactory-Id: redacted:-8000

[CWSpear]

Lastly, we were able to reproduce this on Ubuntu that was not using docker-machine/boot2docker and multiple people's Macs using docker-machine.

[stephengfriend]

@CWSpear: +1, reproduced with boot2docker 1.8.1 + docker 1.8.1

[puja108]

@dmp42 true, might be different.

[jibwa]

+1 reproduced the same problem with docker 1.8.1.

[mkillianey]

+1 We're also experiencing this using 1.8.1 to talk to our internally-hosted Artifactory running 3.6 (which only supports docker v1 repos), whereas pre-1.8 versions of docker work fine.

However, I just discovered that docker pull --all-tags image will load.

[dmp42]

@CWSpear export DOCKER_CONTENT_TRUST=1 will enable trust, which is a v2 feature only, and that will definitely not work without notary.

Everybody here: please, no +1 without any details about your exact configuration.

I just started a registry: docker run -p 5000:5000 registry:latest

Then pushed to it and pulled from it successfully using docker 1.8.1.

If you are experiencing issues:

• if you are not using one of the latest v1 registry (0.8.1 or 0.9.1), please upgrade
• if you are using a third-party registry (redhat, artifactory, or else) please bring it to their support
• if you are using registry 0.8.1 or 0.9.1, please provide your registry logs, and any other relevant details about it (proxy, authentication, and backend storage configuration)

Thanks.

[CWSpear]

@dmp42 has informed me we will not be leavingdocker 1.7, not just in the registry, but in production.

So the next alternative would be to allow docker-machine (and I can open a ticket there if needed) to allow for machines to have different versions of docker, so I can have a VM for 1.7.1 for this client and the latest for my other clients. As far as I can tell, all local machines will share the same boot2docker.iso.

[dmp42]

@CWSpear did you get information or logs from artifactory?

[CWSpear]

No. But I did find out that we're using Artifactory 3.6.0 and I found this link: http://www.jfrog.com/confluence/display/RTF/Artifactory+4.0.2. Support for docker 1.8 was added in that 4.0.2 version.

[dmp42]

Ah thanks a lot for the info @CWSpear

People: if you are using Artifactory see above ^.

[CWSpear]

I found the original ticket which may be helpful to you and others (logging from the client I could never find), and it shows that this fix was applied in the Artifactory 3.9 branch, as well: https://www.jfrog.com/jira/browse/RTFACT-7920

[AkihiroSuda]

v1 support has been deprecated. Let me close this issue.