Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Private registry push fail: server gave HTTP response to HTTPS client #1874

Closed
yuqian0218 opened this issue Aug 1, 2016 · 40 comments

Comments

@yuqian0218
Copy link

commented Aug 1, 2016

My private registry worked well based on docker 1.10.3, but it can't pull/push images after docker updated to 1.12.0.

I had modified the /etc/sysconfig/docker as:
OPTIONS='--selinux-enabled=true --insecure-registry=myip:5000'
or
OPTIONS='--selinux-enabled=true --insecure-registry myip:5000'
but when I exec pull/push,I got this error:
$ docker pull myip:5000/cadvisor
Using default tag: latest
Error response from daemon: Get https://myip:5000/v1/_ping: http: server gave HTTP response to HTTPS client
when I change back docker to 1.10.3, it still work well as below:
$ docker pull myip:5000/cadvisor
Using default tag: latest
Trying to pull repository myip:5000/cadvisor ...
latest: Pulling from myip:5000/cadvisor
09d0220f4043: Pull complete
a3ed95caeb02: Pull complete
151807d34af9: Pull complete
14cd28dce332: Pull complete
Digest:
sha256:33b6475cd5b7646b3748097af1224de3eee3ba7cf5105524d95c0cf135f59b47
Status: Downloaded newer image for myip:5000/cadvisor:latest

As suggested by RichardScothern, some relative informations are listed below:
docker version
Client:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built:
OS/Arch: linux/amd64

Server:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built:
OS/Arch: linux/amd64

docker info
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 241
Server Version: 1.12.0
Storage Driver: devicemapper
Pool Name: docker-253:0-6809-pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 5.459 GB
Data Space Total: 107.4 GB
Data Space Available: 34.74 GB
Metadata Space Used: 9.912 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.138 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use '--storage-opt dm.thinpooldev' to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host overlay null bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 62.39 GiB
Name: server_3
ID: TITS:BL4B:M5FE:CIRO:5SW6:TVIV:HW36:J7OS:WLHF:46T6:2RBA:WCNV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 21
Goroutines: 32
System Time: 2016-08-02T10:33:06.414048675+08:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8

docker exec <registry-container> registry -version
registry github.com/docker/distribution v2.2.1

After I restart the docker daemon in debug mode, the daemon logs when reproducing my problem are listed below:
DEBU[0794] Calling POST /v1.24/images/create?fromImage=10.10.10.40%3A5000%2Fcadvisor&tag=latest
DEBU[0794] hostDir: /etc/docker/certs.d/10.10.10.40:5000
DEBU[0794] hostDir: /etc/docker/certs.d/10.10.10.40:5000
DEBU[0794] Trying to pull 10.10.10.40:5000/cadvisor from https://10.10.10.40:5000 v2
WARN[0794] Error getting v2 registry: Get https://10.10.10.40:5000/v2/: http: server gave HTTP response to HTTPS client
ERRO[0794] Attempting next endpoint for pull after error: Get https://10.10.10.40:5000/v2/: http: server gave HTTP response to HTTPS client
DEBU[0794] Trying to pull 10.10.10.40:5000/cadvisor from https://10.10.10.40:5000 v1
DEBU[0794] hostDir: /etc/docker/certs.d/10.10.10.40:5000
DEBU[0794] attempting v1 ping for registry endpoint https://10.10.10.40:5000/v1/
DEBU[0794] Fallback from error: Get https://10.10.10.40:5000/v1/_ping: http: server gave HTTP response to HTTPS client
ERRO[0794] Attempting next endpoint for pull after error: Get https://10.10.10.40:5000/v1/_ping: http: server gave HTTP response to HTTPS client
ERRO[0794] Handler for POST /v1.24/images/create returned error: Get https://10.10.10.40:5000/v1/_ping: http: server gave HTTP response to HTTPS client
DEBU[1201] clean 2 unused exec commands

What's more, I just run a simple command to launch the private registry for test, anything else is by default:
docker run -d -p 5000:5000 --restart=always --name registry -v 'pwd'/data:/var/lib/registry registry:2
Neither nginx nor proxy is configured. In summary, it is only a quiet sample environment for test.

Hope you guys giving me some suggestions ,thank you!

@RichardScothern

This comment has been minimized.

Copy link
Contributor

commented Aug 1, 2016

Please follow these instructions to help us diagnose your issue

  1. create a new issue, with a succinct title that describes your issue:
    • bad title: "It doesn't work with my docker"
    • good title: "Private registry push fail: 400 error with E_INVALID_DIGEST"
  2. copy the output of:
    • docker version
    • docker info
    • docker exec <registry-container> registry -version
  3. copy the command line you used to launch your Registry
  4. restart your docker daemon in debug mode (add -D to the daemon launch arguments)
  5. reproduce your problem and get your docker daemon logs showing the error
  6. if relevant, copy your registry logs that show the error
  7. provide any relevant detail about your specific Registry configuration (e.g., storage backend used)
  8. indicate if you are using an enterprise proxy, Nginx, or anything else between you and your Registry
@yuqian0218

This comment has been minimized.

Copy link
Author

commented Aug 2, 2016

@RichardScothern ,thank you for your comment and I will modify the issue with your suggestions.

@yuqian0218 yuqian0218 changed the title can't pull/push images after updating docker to 1.12 Private registry push fail: server gave HTTP response to HTTPS client Aug 2, 2016

@RichardScothern

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2016

Can we see your config @wudiapo135 ? Do you have tls configured?

@dmcgowan

This comment has been minimized.

Copy link
Member

commented Aug 2, 2016

Looks like a docker configuration issue. The --insecure-registry=myip:5000 flag is not getting set on the daemon, causing this error. Try running the daemon manually with your desired options and see if you get the same issue.

@yuqian0218

This comment has been minimized.

Copy link
Author

commented Aug 3, 2016

I get helped from [http://stackoverflow.com/questions/38695515/can-not-pull-push-images-after-update-docker-to-1-12], two steps in total to solve this issue:

  1. Create or modify /etc/docker/daemon.json
    { "insecure-registries":["myregistry.example.com:5000"] }
  2. Restart docker daemon
    sudo service docker restart

I agree with @dmcgowan

The--insecure-registry=myip:5000 flag is not getting set on the daemon

but I have no idea about why it only occurred under docker version 1.12. I will keep this issue open in next three days, any comments are welcome.

@yuqian0218

This comment has been minimized.

Copy link
Author

commented Aug 3, 2016

Can we see your config @wudiapo135 ? Do you have tls configured?

I had never change the config for tls, so tls config is setting by default.

@yuqian0218 yuqian0218 closed this Aug 10, 2016

@daniloascione

This comment has been minimized.

Copy link

commented Sep 20, 2016

Same problem here but with Docker for Mac Version 1.12.1-beta26.1 (build: 12100).
Solved adding the insecure registry in Docker Mac App preferences.
Why this issue is closed?

@RichardScothern

This comment has been minimized.

Copy link
Contributor

commented Sep 20, 2016

@daniloascione the OP closed this issue because he fixed the cause of the error by correctly setting the --insecure-registry flag. If you are having a similar issue and this is not helping you then open another issue describing your problem.

@daniloascione

This comment has been minimized.

Copy link

commented Sep 21, 2016

@RichardScothern I see... so the correct way to set the insecure-registry flag is modifying /etc/docker/daemon.json, and the --insecure-registry=myip:5000 flag is not getting set on the daemon, as reported before. Thank you.

@raof01

This comment has been minimized.

Copy link

commented Oct 25, 2016

@wudiapo135, I did the same per your comments, but still got the same error: Private registry push fail: server gave HTTP response to HTTPS client

My docker version: Docker version 1.12.2, build bb80604

@BadNews87

This comment has been minimized.

Copy link

commented Oct 25, 2016

I also have the same problem with this docker version for Mac.

Docker version 1.12.2, build bb80604

registry added to insecure registries in preferences but no luck. Worked in 1.12.1.

@orlade

This comment has been minimized.

Copy link

commented Nov 7, 2016

With Docker For Mac, the registries setting doesn't seems to be very sticky. I originally added my registry as https:// and got this error. I changed the address to http and restarted Docker, but the error persisted.

After removing the setting altogether, restarting Docker, then adding the setting back and restarting again it stuck and started working. YMMV.

@altostratous

This comment has been minimized.

Copy link

commented Feb 6, 2017

I have the same issue with docker 1.12.4.

@xingyuli

This comment has been minimized.

Copy link

commented Feb 15, 2017

Same issue with 1.13.1, solved using solution of @wudiapo135

@BadNews87

This comment has been minimized.

Copy link

commented Feb 15, 2017

I had a Beta version when I've installed the stable version the issue did not appear anymore.

@shadabb2000

This comment has been minimized.

Copy link

commented Mar 3, 2017

I am still getting the issue in version 1.12.6 and running the registry using the command : docker run -d -p 5000:5000 --restart=always --name registry registry:2

@pradeepkumards

This comment has been minimized.

Copy link

commented Mar 9, 2017

i have same issue with docker 1.12.6.

@kcmerrill

This comment has been minimized.

Copy link

commented Mar 10, 2017

Same issue since upgrading, adding --insecure-registry localhost:5000 fixed it(using the docker registry image in my case).

Would be nice to have localhost automagically added ...

@saavkaar

This comment has been minimized.

Copy link

commented Mar 16, 2017

Try adding --insecure-registry option to daemon in /etc/systemd/system/docker.service.d/docker.conf file.
Then sudo systemctl daemon-reload
And sudo service docker restart

It worked for me

OS: Ubuntu 16.04
Docker: 1.26

@danipl

This comment has been minimized.

Copy link

commented Apr 9, 2017

The same problem here, solved modifying the "/etc/docker/daemon.json" file just like @wudiapo135 suggested.

docker -v -> Docker version 17.03.1-ce, build c6d412e
uname -r -> 4.8.0-36-generic (Ubuntu 16.04.2 LTS Xenial Xerus)

@rmorales-iaa

This comment has been minimized.

Copy link

commented Apr 12, 2017

For Centos 7 and Docker version _17.03.1-ce, build c6d412e_
, just modify ' /usr/lib/systemd/system/docker.service', as @saavkaar indicated:

vi /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --insecure-registry 192.168.127.1:5000

And now reset docker:

systemctl daemon-reload
service docker restart

Where 192.168.127.1:5000 if the 'IP:port' of the master node where the registry image is running.
Apply this modification and the restart in the master node and also in the slaves.

Now start the registry image in the master node:
docker run -d -p 5000:5000 --restart=always --name registry -v LOCAL_PATH:/var/lib/registry registry:2

Where LOCAL_PATH is a existent directory in your master node.

Push an image intto your registry before you can pull.

In the master node:
docker push 192.168.127.1:5000:/YOUR_IMAGE

Where YOUR_IMAGE is the name of the image that you want distribute.

Now you can pull

In the slaves nodes:
docker pull 192.168.127.1:5000/:YOUR_IMAGE

@pranay-91

This comment has been minimized.

Copy link

commented Jun 3, 2017

I have an aws ubuntu instance 14.04. I can login via the instance or host but cannot log in from outside.

http: server gave HTTP response to HTTPS client

Is this to do with docker?
Do i need to configure firewall and open ports in aws instance group policy? I have allowed 5000 http 80 and https 443.

bash [Service] EnvironmentFile=/etc/default/docker ExecStart= ExecStart=/usr/bin/docker daemon -H fd:// --insecure-registry ec2-35-160-82-207.us-west-2.compute.amazonaws.com:5000 35.160.82.207:5000
Ubuntu 16.04 aws instance
Docker version 17.03.1-ce, build c6d412e

@rmorales-iaa

This comment has been minimized.

Copy link

commented Jun 5, 2017

Dear @pranay-91,
Check usual things: port 5000 is not in use, firewall configuration and Docker log entries.
User @saavkaar ran it in Ubuntu 16.04 and Docker: 1.26

@rojagit

This comment has been minimized.

Copy link

commented Jul 28, 2017

I had the insecure-registry url already, and while it worked once, it didn't work after a VM restart.
I just had to restart docker and it worked this time without the https gave http error.

@tomasaschan

This comment has been minimized.

Copy link

commented Aug 6, 2017

I had the same problem as here, but with Docker on Windows. Turns out that the file at C:\ProgramData\docker\config\daemon.json isn't the only source of config here; if I right-click the docker icon in the taskbar and choose Settings...->Daemon and enable advanced config editing, I get a different set of settings.

Adding the insecure registry there, not in the daemon.json file on disk, seems to have solved my problem.

@Applemann

This comment has been minimized.

Copy link

commented Aug 17, 2017

Also you can install haproxy and add into config:

frontend http
        bind *:80
        redirect scheme https if !{ ssl_fc }

frontend https
        bind *:443 ssl crt {{{ your certificate  }}}
        acl host_docker hdr(host) -i docker.domain.com
        reqadd X-Forwarded-Port:\ 443
        reqadd X-Forwarded-Proto:\ https
        reqadd X-Forwarded-Scheme:\ https
        use_backend docker if host_docker

backend docker
        reqadd X-Forwarded-Host:\ docker.domain.com
        server docker 127.0.0.1:5000

then you don't need --insecure-registry flag..

@ProProgrammer

This comment has been minimized.

Copy link

commented Aug 22, 2017

Is /etc/docker/daemon.json the correct file path to add {"insecure-registries": ["172.16.231.128:5000"]} on macOS Sierra Version 10.12.6?
docker --version --> Docker version 17.06.0-ce, build 02c1d87

I have private registry running on a Ubuntu VM on my Mac which is accessible via SSH on my Mac
On Ubuntu 14.04 VM:
docker --version --> Docker version 17.06.1-ce, build 874a737

I am able to push to private registry from within the Ubuntu VM, but when I try to push it from Mac using the VM's IP Address (that I use to SSH into my VM), I get the output

The push refers to a repository [172.16.231.128:5000/alpine-mymac]
Get https://172.16.231.128:5000/v2/: http: server gave HTTP response to HTTPS client

Note: 172.16.231.128 is my VM's IP Address

@Applemann

This comment has been minimized.

Copy link

commented Aug 22, 2017

@ProProgrammer on mac OS X is daemon.json in path: ~/Library/Containers/com.docker.docker/Data/database/com.docker.driver.amd64-linux/etc/docker

@ProProgrammer

This comment has been minimized.

Copy link

commented Aug 23, 2017

@Applemann etc/docker does not exist in /Users/ds/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux for me. Should I just create the full path /Users/ds/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/etc/docker/daemon.json?

@Applemann

This comment has been minimized.

Copy link

commented Aug 23, 2017

@ProProgrammer yes you can try it.

@Gmentsik

This comment has been minimized.

Copy link

commented Sep 5, 2017

This is a mess, we have the same problem, it worked fine until now.

I have added the /etc/docker/daemon.json before and it worked just fine....

How can we correct his issue?

Btw: I'm using Gitlab as my registy...

@zenakuten

This comment has been minimized.

Copy link

commented Oct 22, 2017

If you are using Docker for Windows with linux containers, the 'insecure-registries' setting is here:
C:\Program Files\Docker\Docker\resources\linux-daemon-options.json

I'm using Docker for Windows, but I'm not actually using the 'for Windows' part. Instead I followed the 'hyperv' instructions.

  1. Install docker for windows but uncheck the 'start at login' box. Instead, follow the instructions for creating a docker machine using hyperv - https://docs.docker.com/machine/drivers/hyper-v/ . I called mine 'dockervm'. I also created a virtual switch that is bridged so it has a real (external) IP.

  2. Create a scheduled task to run at startup 'C:\Program Files\Docker\Docker\resources\bin\docker-machine start dockervm'. Make sure to not use double quotes as there is a bug in the windows 10 task scheduler.

  3. After step 1 your docker machine is running, use 'docker-machine env dockervm' to get the environment, and set it in your global environment settings.

  4. After a reboot, your dockervm should be running, and docker ps -a should return results.

  5. Run the registry locally: docker run -d -p 5000:5000 --name registry registry:2

  6. Open 'Hyper-V Manager' and select 'dockervm' (it should be running). Click 'Connect...' under dockervm on the right to open a shell. You should now be at a root shell prompt in your dockervm

  7. From the root shell prompt, cd to /var/lib/boot2docker

  8. vi profile

Add a new line to this part with your registry (my vm's IP is 192.168.1.24)

EXTRA_ARGS='
--label provider=hyperv
--insecure-registry=192.168.1.24:5000
'
8. Restart the dockervm machine in hyperv manager

You should now be able to push to the registry

@ghost

This comment has been minimized.

Copy link

commented Dec 18, 2017

I'd like to clarify that you should add the { "insecure-registries":["myregistry.example.com:5000"] } to /etc/docker/daemon.json in the client machine.

@emailtovinod

This comment has been minimized.

Copy link

commented Dec 29, 2017

I added the docker registry network CIDR in '/etc/sysconfig/docker ' of clients eg: 'OPTIONS= - --insecure-registry=192.168.0.0/24' and is working fine.

@FlorisE

This comment has been minimized.

Copy link

commented May 28, 2018

For Mac users, it seems like they added the ability to configure insecure registries in the GUI, via Preferences > Daemon > Insecure registries.

@arno01

This comment has been minimized.

Copy link

commented Jul 10, 2018

tl;dr make sure you are using a correct user/password :-)

According to the https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry - message http: server gave HTTP response to HTTPS client does not mean it is a reason of the failure:

With insecure registries enabled, Docker goes through the following steps:

First, try using HTTPS.

  • If HTTPS is available but the certificate is invalid, ignore the error about the certificate.
  • If HTTPS is not available, fall back to HTTP.

You can observe this message in both, working (correct password) & non-working (incorrect password) examples below:

docker login

$ echo corr3ctpassword | docker login -u user --password-stdin reg.mysite.com:80
Login Succeeded

$ journalctl -u docker -f
Jul 10 21:34:19 laptop dockerd[30795]: time="2018-07-10T21:34:19.365956391+02:00" level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://reg.mysite.com:80/v2/: http: server gave HTTP response to HTTPS client"

docker login with the wrong password

$ echo wr0ngpassword | docker login -u user --password-stdin reg.mysite.com:80
Error response from daemon: Get http://reg.mysite.com:80/v2/: unauthorized: authentication required

$ journalctl -u docker -f
Jul 10 21:33:38 laptop dockerd[30795]: time="2018-07-10T21:33:38.200034001+02:00" level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://reg.mysite.com:80/v2/: http: server gave HTTP response to HTTPS client"

Setup

docker version: 17.12.1-ce

Either A. or B., both cannot work together.

When you change docker.service files, do not forget to run systemctl daemon-reload && systemctl restart docker as root.

  • A) /etc/systemd/system/docker.service.d/insecure.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry reg.mysite.com:80
  • B) /etc/docker/daemon.json
{
  "insecure-registries" : ["reg.mysite.com:80"]
}
@Forsk

This comment has been minimized.

Copy link

commented Sep 26, 2018

ubuntu
Edit configuration file /etc/systemd/system/multi-user.target.wants/docker.service
add ExecStart=/usr/************* --insecure-registry yourip:5000

@ghost

This comment has been minimized.

Copy link

commented Dec 13, 2018

i added ExecStart=/usr/************* --insecure-registry yourip:5000 ,but still can't work, the same as {
"insecure-registries" : ["reg.mysite.com:80"]
}

@nayrangnu

This comment has been minimized.

Copy link

commented Jan 16, 2019

For future people who had my problem:
If you installed docker using snap (run snap services to check if docker.dockerd is listed), you will need to add the insecure-registries entry to /var/snap/docker/current/config/daemon.json, not the default config location.

@santhoshkumarhirekerur

This comment has been minimized.

Copy link

commented Feb 28, 2019

For Docker version 18.09.2, I followed https://success.docker.com/article/using-systemd-to-control-the-docker-daemon

  1. sudo systemctl edit docker
  2. add below lines

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry registry:5000

  1. sudo systemctl daemon-reload
  2. systemctl restart docker
  3. systemctl status docker
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.