Airgap - Rancher container cannot recognize the private registry

[jiaqiluo]

What kind of request is this (question/bug/enhancement/feature request):
bug

Steps to reproduce (least amount of steps as possible):

• run rancher:v2.5.0-alpha3 single install in airgap env

sudo docker run -d --privileged --restart=unless-stopped -p 80:80 -p 443:443 -v ${PWD}/fullchain.pem:/etc/rancher/ssl/cert.pem -v ${PWD}/privkey.pem:/etc/rancher/ssl/key.pem -e CATTLE_SYSTEM_DEFAULT_REGISTRY=ec2-18-191-239-133.us-east-2.compute.amazonaws.com -e CATTLE_SYSTEM_CATALOG=bundled ec2-18-191-239-133.us-east-2.compute.amazonaws.com/rancher/rancher:v2.5.0-alpha3 --no-cacerts

• check the logs of the container

Result:

E0917 18:57:08.364148 23 pod_workers.go:191] Error syncing pod 0bf36ee9-6ccf-4c6a-9779-2ebb241257fd ("helm-operation-nrjhs_cattle-system(0bf36ee9-6ccf-4c6a-9779-2ebb241257fd)"), skipping: [failed to "StartContainer" for "helm" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"ec2-18-191-239-133.us-east-2.compute.amazonaws.com/rancher/shell:v0.1.2\": failed to resolve reference \"ec2-18-191-239-133.us-east-2.compute.amazonaws.com/rancher/shell:v0.1.2\": failed to do request: Head https://ec2-18-191-239-133.us-east-2.compute.amazonaws.com/v2/rancher/shell/manifests/v0.1.2: x509: certificate signed by unknown authority", failed to "StartContainer" for "proxy" with ImagePullBackOff: "Back-off pulling image \"ec2-18-191-239-133.us-east-2.compute.amazonaws.com/rancher/shell:v0.1.2\""]

[maggieliu]

Need to document this in Rancher docs and also add to the release notes. If there are certs on the registry need to follow the k3s doc for adding the private-registry.

The certs and registry confirmation files need to be mounted into Rancher container.

https://rancher.com/docs/k3s/latest/en/installation/private-registry/

[lushtech]

@maggieliu

I resolved the problem using the method above you mentioned in the rancher master cluster after a SLB ,and the local cluster

work well. The ECS is no public IP and use private registry.

But when I deployed another cluster ,the problem comes again. This cluster can use private chart registry and deployed the

apps. The ECS is no public IP and use private registry.

but I can not use kubectl in the UI in this cluster. The error message is showed below:

Failed to pull image "rancher/shell:v0.1.5": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

and I didnot find the way to mount the certs and registry confirmation files as start rancher master cluster when I deployed the cluster use ui in the rancher.
why the other pods know pull images from private registry ,but rancher/shell:v0.1.5 that can not?

[lushtech]

@maggieliu

I resolved the problem using the method above you mentioned in the rancher master cluster after a SLB ,and the local cluster

work well. The ECS is no public IP and use private registry.

But when I deployed another cluster ,the problem comes again. This cluster can use private chart registry and deployed the

apps. The ECS is no public IP and use private registry.

but I can not use kubectl in the UI in this cluster. The error message is showed below:

Failed to pull image "rancher/shell:v0.1.5": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

and I didnot find the way to mount the certs and registry confirmation files as start rancher master cluster when I deployed the cluster use ui in the rancher.
why the other pods know pull images from private registry ,but rancher/shell:v0.1.5 that can not?

I fix this by add the private registry in the setting of system-default-registry in Global Tab in the rancher UI.

Hope this can be help to others.

[SheilaghM]

Pull k3s docs into Rancher docs and have QA review our docs. Private Registry is not well documented in Docker, and we don't cover it in Rancher.

[vineetkumar03]

@maggieliu
Thanks for reply .. but it will work when rancher creation without bind mount on /var/lib/rancher . For Ex:
docker run -d --restart=unless-stopped --privileged -p 80:80 -p 443:443 rancher/rancher:v2.5.5
If u want to run rancher with bind mount with registry.yaml and local registry image For Ex
docker run -d --restart=unless-stopped --privileged --name rancher -p 80:80 -p 443:443 -v /opt/rancher:/var/lib/rancher -v /home2/secrets:/etc/rancher/k3s REGISTRY.DOMAIN.COM:PORT/rancher/rancher:v2.5.5
registry.yaml and pem files resides under /home2/secrets directory
For Ref:
https://vineetcic.medium.com/air-gap-installation-of-single-node-rancher-13f705e16429
https://rancher.com/docs/rancher/v2.5/en/installation/other-installation-methods/single-node-docker/advanced/#air-gap
https://rancher.com/docs/k3s/latest/en/installation/private-registry/