Private registry push fail: server gave HTTP response to HTTPS client

[ghost]

Hi, I am trying to get hands dirty with container technology, and this is first time I am trying to set up docker.
I was trying to push an Image (Ubuntu) to my private registry, when I got this error.
Already went through multiple threads ( #1929, #26781, #1874), none can help me out yet.
Here are some relevant Information that might help you understand the issue:

Docker Info

[root@cent7 ~]# docker info
Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 3
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-253:0-223687-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 241.6 MB
 Data Space Total: 107.4 GB
 Data Space Available: 5.563 GB
 Metadata Space Used: 913.4 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay bridge host null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993.3 MiB
Name: cent7
ID: PGIC:2YZN:YMYG:NGA7:F7W2:3IEX:P2TN:ZINS:J7EP:FBIZ:EWGH:EHLY
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://10.136.8.10:8080/
No Proxy: localhost,127.0.0.1,cent7:5000
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Docker Version

[root@cent7 ~]# docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        
 OS/Arch:      linux/amd64

Docker Registry

[root@cent7 ~]# docker exec amit_reg registry --version
registry github.com/docker/distribution v2.5.1

Some extra info.

• The network which I am at uses a proxy server, so I updated the file: /etc/systemd/system/docker.service.d/http-proxy.conf

[root@cent7 ~]# cat /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://10.136.8.10:8080/" "NO_PROXY=localhost,127.0.0.1,cent7:5000"

• Also, Since I want my registry to be insecure, Updated the file: etc/systemd/system/docker.service.d/docker.conf

[root@cent7 docker.service.d]# cat /etc/systemd/system/docker.service.d/docker.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --insecure-registry cent7:5000

But that resulted in the following error:

[root@cent7 docker.service.d]# systemctl restart docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

So commented it and decided to use environment variables instead.

• In the file /usr/lib/systemd/system/docker.service, set path for EnvironmentFile

[root@cent7 docker.service.d]# cat /usr/lib/systemd/system/docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target

[Service]
EnvironmentFile=-/etc/sysconfig/docker
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

• In environment file added --insecure-registry. (Source: this blog entry)

[root@cent7 docker.service.d]# cat /etc/sysconfig/docker 
other_args="--insecure-registry cent7:5000"

• And now I am stuck at this issue

[root@cent7 ~]# docker push cent7:5000/ubuntu
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

• The debug log for docker:

INFO[0000] API listen on /var/run/docker.sock           
DEBU[0011] Calling GET /v1.24/containers/json  # docker ps          
DEBU[0018] Calling GET /v1.24/images/json       # docker images
# Here docker push was run. [docker push cent7:5000/ubuntu] 
DEBU[0025] Calling POST /v1.24/images/cent7:5000/ubuntu/push?tag=  
DEBU[0025] hostDir: /etc/docker/certs.d/cent7:5000      
DEBU[0025] hostDir: /etc/docker/certs.d/cent7:5000      
DEBU[0025] Trying to push cent7:5000/ubuntu to https://cent7:5000 v2 
DEBU[0025] Error getting v2 registry: Get https://cent7:5000/v2/: http: server gave HTTP response to HTTPS client 
ERRO[0025] Attempting next endpoint for push after error: Get https://cent7:5000/v2/: http: server gave HTTP response to HTTPS client 
DEBU[0025] Trying to push cent7:5000/ubuntu to https://cent7:5000 v1 
DEBU[0025] hostDir: /etc/docker/certs.d/cent7:5000      
DEBU[0025] attempting v1 ping for registry endpoint https://cent7:5000/v1/ 
ERRO[0025] Attempting next endpoint for push after error: Get https://cent7:5000/v1/_ping: http: server gave HTTP response to HTTPS client

[RichardScothern]

Can you ps on the machine your registry is running on and ensure that --insecure-registry is set on the registry process.

Can you curl the registry cent7:5000/v2/ from the host you are running the daemon on?

When you attempt to contact the registry, do you see your requests being logged?

[ghost]

Thank you Richard. The issue is resolved now. I realized that I set the environment variable (to set --insecure-registry) in a wrong format. And also the environment variable name was not matching with the file in service.docker.d/docker.conf, After some hit and trials reading multiple documents, I was able to get it right. I recently switched to rhel7 from 6, that might have caused the problem. I appreciate your time. Thanks. Amit

[saavkaar]

Try adding --insecure-registry option to daemon in /etc/systemd/system/docker.service.d/docker.conf file.
Then sudo systemctl daemon-reload
And sudo service docker restart

It worked for me

OS: Ubuntu 16.04
Docker: 1.26