New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rancher server cannot be launched with no internet access (private registry approach) #8512

Closed
pstrzelczak opened this Issue Apr 14, 2017 · 9 comments

Comments

Projects
None yet
8 participants
@pstrzelczak

pstrzelczak commented Apr 14, 2017

Rancher Versions:
Server: v1.5.3

Docker Version:
Docker version 1.12.6, build 96d83a5/1.12.6

OS and where are the hosts located? (cloud, bare metal, etc):
CentOS 7.2 VM

Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)
single node rancher

Environment Type: (Cattle/Kubernetes/Swarm/Mesos)
Cattle

Steps to Reproduce:
I generally followed the procedure given in https://docs.rancher.com/rancher/v1.5/en/installing-rancher/installing-server/no-internet-access/

  1. Setup local private registry (localhost:5000) with rancher docker images I found started in regular on-line install of rancher (see below for listing)
  2. Disable outgoing connections to ports 80/443 via iptables
  3. Start Rancher master via docker run -d --restart=unless-stopped -p 8080:8080 -e CATTLE_BOOTSTRAP_REQUIRED_IMAGE=localhost:5000/rancher/agent:v1.2.1 localhost:5000/rancher/server:v1.5.3
  4. Add local registry to Rancher via GUI and make it default via registry.default=localhost:5000
  5. Add local host viadocker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher localhost:5000/rancher/agent:v1.2.1 http://10.9.4.75:8080/v1/scripts/CE2EBC9C7F1E9A1506B0:1483142400000:66mc7gVyYC3BJRTAqLa6kivo23o

Results:

Host is not added to Rancher as expected.
I checked syslog and found the agent was trying to pull from docker.io instead of local registry.

Additional information

[xxx@localhost ~]$ docker images
REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
docker.io/registry                       2                   136c8b16df20        7 days ago          33.17 MB
localhost:5000/rancher/net               holder              665d9f6e8cc1        9 days ago          267.2 MB
localhost:5000/rancher/server            v1.5.3              c92aceac67b4        2 weeks ago         935.7 MB
localhost:5000/rancher/network-manager   v0.5.3              0f224908d730        3 weeks ago         241.6 MB
localhost:5000/rancher/metadata          v0.8.11             19b37bb3e242        6 weeks ago         251.5 MB
localhost:5000/rancher/agent             v1.2.1              9cecf992679f        6 weeks ago         233.7 MB
localhost:5000/rancher/net               v0.9.4              5ac4ae5d7fa4        6 weeks ago         264.3 MB
localhost:5000/rancher/dns               v0.14.1             4e37fc4150c2        7 weeks ago         239.8 MB
localhost:5000/rancher/healthcheck       v0.2.3              491349141109        11 weeks ago        383.3 MB
[xxx@localhost ~]$ docker ps
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
e22aaa5d18a8        localhost:5000/rancher/agent:v1.2.1    "/run.sh run"            17 minutes ago      Up 17 minutes                                          rancher-agent
ee97d97c24c4        localhost:5000/rancher/server:v1.5.3   "/usr/bin/entry /usr/"   20 minutes ago      Up 20 minutes       3306/tcp, 0.0.0.0:8080->8080/tcp   determined_colden
2ce204d224b3        registry:2                             "/entrypoint.sh /etc/"   23 minutes ago      Up 23 minutes       0.0.0.0:5000->5000/tcp             docker-registry

/var/log/messages

Apr 14 07:18:56 localhost journal: time="2017-04-14T11:18:56Z" level=error msg="Error processing event" err="Image [index.docker.io/rancher/healthcheck:v0.2.3] failed to pull: Pulling repository docker.io/rancher/healthcheck" eventId=1d6f99f2-4047-418d-a939-6f1312c787ca eventName=storage.image.activate resourceId=1ispm35 
Apr 14 07:18:56 localhost journal: 2017-04-14 11:18:56,696 ERROR [b7f1f9f5-f1a1-4c8b-978c-c5fb81197554:897] [instance:48->imageStoragePoolMap:35] [instance.start->(InstanceStart)->volume.activate->(VolumeActivate)->imagestoragepoolmap.activate] [] [cutorService-19] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [storage.image.activate.reply;agent=1]: Image [index.docker.io/rancher/healthcheck:v0.2.3] failed to pull: Pulling repository docker.io/rancher/healthcheck 
Apr 14 07:18:56 localhost journal: 2017-04-14 11:18:56,696 ERROR [b7f1f9f5-f1a1-4c8b-978c-c5fb81197554:897] [instance:48->volume:66] [instance.start->(InstanceStart)->volume.activate] [] [cutorService-19] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [storage.image.activate.reply;agent=1]: Image [index.docker.io/rancher/healthcheck:v0.2.3] failed to pull: Pulling repository docker.io/rancher/healthcheck 
Apr 14 07:18:56 localhost journal: 2017-04-14 11:18:56,696 ERROR [b7f1f9f5-f1a1-4c8b-978c-c5fb81197554:897] [instance:48] [instance.start->(InstanceStart)] [] [cutorService-19] [i.c.p.process.instance.InstanceStart] Failed to Storage for instance [48] 
Apr 14 07:18:56 localhost journal: 2017-04-14 11:18:56,710 ERROR [b7f1f9f5-f1a1-4c8b-978c-c5fb81197554:897] [instance:48] [instance.start] [] [cutorService-19] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [storage.image.activate.reply;agent=1]: Image [index.docker.io/rancher/healthcheck:v0.2.3] failed to pull: Pulling repository docker.io/rancher/healthcheck 
Apr 14 07:18:56 localhost journal: 2017-04-14 11:18:56,732 ERROR [:] [] [] [] [cutorService-19] [.e.s.i.ProcessInstanceDispatcherImpl] Agent error for [storage.image.activate.reply;agent=1]: Image [index.docker.io/rancher/healthcheck:v0.2.3] failed to pull: Pulling repository docker.io/rancher/healthcheck 
@aemneina

This comment has been minimized.

Show comment
Hide comment
@aemneina

aemneina Apr 18, 2017

is your agent on the same host as the server?
edit: n.m. seems that's the case.

aemneina commented Apr 18, 2017

is your agent on the same host as the server?
edit: n.m. seems that's the case.

@aemneina

This comment has been minimized.

Show comment
Hide comment
@aemneina

aemneina Apr 18, 2017

Did you add the host to the default environment? I believe after changing registry.default, only newly created environments will pull that setting down. Can you check that?

aemneina commented Apr 18, 2017

Did you add the host to the default environment? I believe after changing registry.default, only newly created environments will pull that setting down. Can you check that?

@gawalivaibhav

This comment has been minimized.

Show comment
Hide comment
@gawalivaibhav

gawalivaibhav Apr 19, 2017

You can point your "index.docker.io" to your local host by editing "/etc/hosts" file.
You just have to add entry like " 127.0.01 index.docker.io".
or " < LAN IP > index.docker.io"
After this index.docker.io will point your localhost/LAN IP

gawalivaibhav commented Apr 19, 2017

You can point your "index.docker.io" to your local host by editing "/etc/hosts" file.
You just have to add entry like " 127.0.01 index.docker.io".
or " < LAN IP > index.docker.io"
After this index.docker.io will point your localhost/LAN IP

@pstrzelczak

This comment has been minimized.

Show comment
Hide comment
@pstrzelczak

pstrzelczak Apr 20, 2017

@aemneina indeed host addition to newly created environment after making my private registry default worked.
Still I would expect to make it possible to add hosts to default environment with no internet too.

pstrzelczak commented Apr 20, 2017

@aemneina indeed host addition to newly created environment after making my private registry default worked.
Still I would expect to make it possible to add hosts to default environment with no internet too.

@deniseschannon

This comment has been minimized.

Show comment
Hide comment
@deniseschannon

deniseschannon Apr 21, 2017

Member

You can delete the infra services in the default environment and then re-launch them through the Catalog (Under Catalog -> Library), which will start the services using your default registry.

Member

deniseschannon commented Apr 21, 2017

You can delete the infra services in the default environment and then re-launch them through the Catalog (Under Catalog -> Library), which will start the services using your default registry.

@sliard

This comment has been minimized.

Show comment
Hide comment
@sliard

sliard May 18, 2017

failed to pull: Pulling repository docker.io/rancher/healthcheck

Rancher don't use you local repository to pull image

In my case I have the same problem if I don't create a new environment after change registry.default properties

I had one other problem with ssl and local docker registry. If you use localhost to test repository, it's work over http, but if you need to use a hostname, you need use https.

sliard commented May 18, 2017

failed to pull: Pulling repository docker.io/rancher/healthcheck

Rancher don't use you local repository to pull image

In my case I have the same problem if I don't create a new environment after change registry.default properties

I had one other problem with ssl and local docker registry. If you use localhost to test repository, it's work over http, but if you need to use a hostname, you need use https.

@superseb

This comment has been minimized.

Show comment
Hide comment
@superseb

superseb Oct 5, 2017

Member

Please comment if there is more help needed on this, we have more documentation on this now: http://rancher.com/docs/rancher/v1.6/en/installing-rancher/installing-server/no-internet-access/#using-a-private-registry

Member

superseb commented Oct 5, 2017

Please comment if there is more help needed on this, we have more documentation on this now: http://rancher.com/docs/rancher/v1.6/en/installing-rancher/installing-server/no-internet-access/#using-a-private-registry

@superseb superseb closed this Oct 5, 2017

@stefanvangastel

This comment has been minimized.

Show comment
Hide comment
@stefanvangastel

stefanvangastel Oct 29, 2017

@superseb How does this apply to Rancher-2.0 since you can't create new environments there I get stuck after setting the registry.default setting.

stefanvangastel commented Oct 29, 2017

@superseb How does this apply to Rancher-2.0 since you can't create new environments there I get stuck after setting the registry.default setting.

@vincent99

This comment has been minimized.

Show comment
Hide comment
@vincent99

vincent99 Oct 29, 2017

Member

Running the 2.0 preview without internet access is not currently supported. It will be added near the end on the release cycle.

Member

vincent99 commented Oct 29, 2017

Running the 2.0 preview without internet access is not currently supported. It will be added near the end on the release cycle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment