Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Config option to reference private registry #160

Closed
chalky9909 opened this issue Nov 23, 2017 · 26 comments

Comments

@chalky9909
Copy link

commented Nov 23, 2017

We are attempting to deploy sonobuoy on a k8s cluster configured behind a firewall with no internet access.

The sonobuoy images are available in a private registry and quickstart.yaml updated to reflect this. When launching the deployment, the sonobuoy specific pods launch with no issues.

However, after a period of time, the below pods fails to launch in the newly created "e2e-tests" namespace.

A kubectl describe against this:
kubectl describe pod pod-submit-remove-16bb7011-d055-11e7-ba38-0a580a02306e -n=e2e-tests-pods-jgshk

shows that it is failing to pull an nginx image as below:

Failed to pull image "gcr.io/google-containers/nginx-slim-amd64:0.20": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 173.194.69.82:443: i/o timeout

As the cluster does not have internet access, it is unable to pull and access this image from gcr.io

This nginx image from gcr.io is not referenced in the deployment yaml so is it possible to add a configuration option where the nginx image can be accessed from a private internal registry?

Something like the below if this is feasible:
private-registry:443/gcr.io/google-containers/nginx-slim-amd64:0.20

Any assistance with this would be much appreciated.

@timothysc

This comment has been minimized.

Copy link
Member

commented Nov 27, 2017

It's been on the radar for a while to help support air-gapped environments. The problem is sifting through all the images required to build and adding an option for the generated .yaml.

@chalky9909

This comment has been minimized.

Copy link
Author

commented Nov 27, 2017

@timothysc thanks for taking a look at this request. I take it then that it is not just the image that is being referred to as below and there are others that are required:

Failed to pull image "gcr.io/google-containers/nginx-slim-amd64:0.20"

@timothysc

This comment has been minimized.

Copy link
Member

commented Nov 27, 2017

@chalky9909 - There are several different images.

@liztio

This comment has been minimized.

Copy link
Contributor

commented Dec 4, 2017

@chalky9909 I'm working on some changes to the the tests to make it feasible to run them on airgapped clusters. Unfortunately it requires upstream changes, and everything is currently in lockdown for the 1.9 release. Once that clears, I should be able to get you an update.

Watch this space!

@chalky9909

This comment has been minimized.

Copy link
Author

commented Dec 7, 2017

@liztio thanks for the update. Will be happy to test once the upstream changes have been made.

@erwbgy

This comment has been minimized.

Copy link

commented Mar 14, 2018

@liztio can you provide a link to the upstream issue?

@mweigel

This comment has been minimized.

Copy link

commented Apr 1, 2018

@liztio I'd also be happy to test this once ready, thanks.

@lewismarshall

This comment has been minimized.

Copy link

commented Apr 10, 2018

@liztio I've needed to run this myself and I'de started on a PR and didn't see your comments. Is it too late for me to share my WIP for sonobouy and the upstream e2e.test binary?

Also I'm having trouble finding the source for the image gcr.io/heptio-images/heptio-e2e:master

@timothysc

This comment has been minimized.

Copy link
Member

commented Apr 10, 2018

@lewismarshall We have not yet open-sourced our extended tests

/cc @kstewart @jbeda

@timothysc timothysc assigned timothysc and unassigned liztio Jun 22, 2018

@timothysc timothysc added this to the v1.11-upstream sync milestone Jun 22, 2018

@chuckha chuckha added the blocked label Jul 5, 2018

@chuckha

This comment has been minimized.

Copy link
Member

commented Jul 11, 2018

Are we punting this out of the milestone or is there something actionable we can do here?

@timothysc

This comment has been minimized.

Copy link
Member

commented Jul 11, 2018

Given the state of upstream I'm fine with punting to 1.0

@timothysc timothysc modified the milestones: v1.11-upstream sync , v1.0.0 Jul 11, 2018

@timothysc

This comment has been minimized.

Copy link
Member

commented Jul 13, 2018

Dropping this breadcrumb here when we loop back to this - kubernetes/kubernetes#38067

@timothysc

This comment has been minimized.

Copy link
Member

commented Aug 23, 2018

ppcle64 conformance is all green. i am working with arm folks now. and trying to get s390x folks started
issues/67721 and issues/67720 in k/k main repo

dims [2:45 PM]
@craigtracey if you want the list of images used, see https://gist.github.com/dims/067c6723588a585a7a00b22e41b86f34

if you find others not on that list when you try the airgapped scenario, please let me know. we will need to update the image and add it to k8s.io/kubernetes/test/utils/image/manifest.go

craigtracey [4:01 PM]
@dims this is great, thank you!
we should probably add this as a command line feature for sonobuoy

timothysc [4:02 PM]
we added an image pull to kubeadm
~=

craigtracey [4:02 PM]
an image pull?


dims [4:05 PM]
it’s a command to list images that are needed by kubeadm `kubeadm config images list`

timothysc [4:07 PM]
`kubeadm config images pull`

craigtracey [4:07 PM]
we are saying the same thing...just words
@timothysc

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

xref - kubernetes/kubernetes#67964

All other multi-arch manifest moves should be complete in 1.12

@timothysc timothysc removed the fixupstream label Sep 21, 2018

@timothysc timothysc removed this from the v1.0.0 milestone Sep 21, 2018

@timothysc timothysc added this to the v1.12-upstream sync milestone Sep 21, 2018

@zbindenren

This comment has been minimized.

Copy link

commented Oct 30, 2018

This feature got merged in master.

It is now possible to reference a yaml file with KUBE_TEST_REPO_LIST environment variable. In this file it is possible to override the environment registries:

dockerLibraryRegistry: docker.io/library
e2eRegistry: gcr.io/kubernetes-e2e-test-images
gcRegistry: k8s.gcr.io
privateRegistry: gcr.io/k8s-authenticated-test
sampleRegistry: gcr.io/google-samples

Since 0.12 is already released, I wonder if this feature still coming to 0.12.x?

@timothysc

This comment has been minimized.

Copy link
Member

commented Oct 30, 2018

@zbindenren the ability to obtain the manifests exists, but the plumbing to override through the e2e tests upstreams for all the different images does not. I will open an issue to cross reference.

@ashishapy

This comment has been minimized.

Copy link

commented Dec 24, 2018

This issue is blocker for any deployment in corporate env. I guess, it is asking a lot from a open source project, any ETA on this.
Thanks

@xdrus

This comment has been minimized.

Copy link

commented Jan 3, 2019

Edit: I found a list of required images here and was able to run tests in isolated environment by pre-downloading these images to worker nodes (there are few missing in this list though, like nginx:latest).

@joshrosso

This comment has been minimized.

Copy link
Member

commented Jan 9, 2019

@timothysc or @liztio, is what's below an accurate summary of this issue's status?

  1. With kubernetes/kubernetes#60848 in 13.0+, using private repositories in E2E tests is possible.

  2. We need to update the plumbing between Sonobuoy and the E2E tests to resolve this issue (#160).

  3. Until then, there are 2 workarounds.

    3a. Pre-load images needed by E2E tests on all worker nodes.

    3b. Write a MutatingWebhook admission controller that alters the image: field on E2E-test pods to use a private repository.

Thanks in advanced!

@erwbgy

This comment has been minimized.

Copy link

commented Jan 9, 2019

We successfully use 3b on-prem.

@timothysc timothysc removed this from the v1.12-upstream sync milestone Jan 11, 2019

@johnSchnake

This comment has been minimized.

Copy link
Contributor

commented Feb 16, 2019

@joshrosso That seems to be my understanding but I haven't (yet) put it to the test. I'm going to see if I can create an airgapped cluster/private registry and really get tests to run that way. I wonder if we are going to find that there are some tests that still reference images directly in the default registries.

@joshrosso

This comment has been minimized.

Copy link
Member

commented Feb 16, 2019

@johnSchnake, i did an airgapped run using 3a on 1.13.3.

There were a few tests that referenced nginx:latest and 2 that had imagePullPolicy: Always. The latter was the more problematic (as a retag of the versioned nginx image to latest worked around the former).

Hope this helps if you go down a similar path.

@stevesloka

This comment has been minimized.

Copy link
Member

commented Feb 20, 2019

xref #593

@stevesloka stevesloka added this to the v0.14 milestone Feb 20, 2019

@johnSchnake

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2019

Working on the solution based on (1) above. I'll put up a PR soon. Manually I was able to just update the yaml to create a configmap with the repo information, mount it into the kube-conformance image, and set an env var to tell k8s to use it.

My test was technically against a public (but custom) registry which seems to be the bigger concern to get airgap to work. Not sure if pulling from a private registry (auth required) going to be different or a hard requirement for this ticket. IMO they seem like they are separate concerns.

@stevesloka

This comment has been minimized.

Copy link
Member

commented Feb 27, 2019

I think a public registry is fine. Did you hack up your /etc/hosts to make sure that docker.io & gcr.io weren't getting called? I guess you could look at the images on a node when the tests are complete to verify as well. I have some PR's coming that will help with testing this process (e.g. image list, image load, etc).

@johnSchnake

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2019

The latter is what I did, just verifying the images that were pulled. I first actually tested against an empty registry to confirm it wouldn't work (and the test container was failing due to the image pull issue) and then pushed the image up and watched the test pass.

The reason I did it this way was so that I could use docker.io but my own custom images (so I was actually using docker.io/schnake instead of /library. This drastically simplified the setup since I was still having problems (and debated the value of) setting up my own public registry which I could get my KIND cluster to trust and which I could load with images.

johnSchnake added a commit to johnSchnake/sonobuoy that referenced this issue Feb 27, 2019

Add support for KUBE_TEST_REPO_LIST and custom registries
K8s 1.13 organized and allowed overrides for where to pull
e2e images from. The env var needed is KUBE_TEST_REPO_LIST.

To facilitate users providing this, the following changes
have been made:

 - the user can specify a file path on the command line with
the proper contents
 - the file is read
 - the contents are used to create a configmap of the values
 - the configmap is mounted as a volume for the kube-conformance
container
 - the env var KUBE_TEST_REPO_LIST is set to point to this file

Fixes heptio#160

johnSchnake added a commit to johnSchnake/sonobuoy that referenced this issue Feb 28, 2019

Add support for KUBE_TEST_REPO_LIST and custom registries
K8s 1.13 organized and allowed overrides for where to pull
e2e images from. The env var needed is KUBE_TEST_REPO_LIST.

To facilitate users providing this, the following changes
have been made:

 - the user can specify a file path on the command line with
the proper contents
 - the file is read
 - the contents are used to create a configmap of the values
 - the configmap is mounted as a volume for the kube-conformance
container
 - the env var KUBE_TEST_REPO_LIST is set to point to this file

Fixes heptio#160

johnSchnake added a commit to johnSchnake/sonobuoy that referenced this issue Feb 28, 2019

Add support for KUBE_TEST_REPO_LIST and custom registries
K8s 1.13 organized and allowed overrides for where to pull
e2e images from. The env var needed is KUBE_TEST_REPO_LIST.

To facilitate users providing this, the following changes
have been made:

 - the user can specify a file path on the command line with
the proper contents
 - the file is read
 - the contents are used to create a configmap of the values
 - the configmap is mounted as a volume for the kube-conformance
container
 - the env var KUBE_TEST_REPO_LIST is set to point to this file

Fixes heptio#160

Signed-off-by: John Schnake <jschnake@vmware.com>

johnSchnake added a commit to johnSchnake/sonobuoy that referenced this issue Feb 28, 2019

Add support for KUBE_TEST_REPO_LIST and custom registries
K8s 1.13 organized and allowed overrides for where to pull
e2e images from. The env var needed is KUBE_TEST_REPO_LIST.

To facilitate users providing this, the following changes
have been made:

 - the user can specify a file path on the command line with
the proper contents
 - the file is read
 - the contents are used to create a configmap of the values
 - the configmap is mounted as a volume for the kube-conformance
container
 - the env var KUBE_TEST_REPO_LIST is set to point to this file

Fixes heptio#160

Signed-off-by: John Schnake <jschnake@vmware.com>

johnSchnake added a commit to johnSchnake/sonobuoy that referenced this issue Feb 28, 2019

Add support for KUBE_TEST_REPO_LIST and custom registries
K8s 1.13 organized and allowed overrides for where to pull
e2e images from. The env var needed is KUBE_TEST_REPO_LIST.

To facilitate users providing this, the following changes
have been made:

 - the user can specify a file path on the command line with
the proper contents
 - the file is read
 - the contents are used to create a configmap of the values
 - the configmap is mounted as a volume for the kube-conformance
container
 - the env var KUBE_TEST_REPO_LIST is set to point to this file

Fixes heptio#160

Signed-off-by: John Schnake <jschnake@vmware.com>

ekj1711 added a commit to ekj1711/sonobuoy that referenced this issue Jul 15, 2019

Add support for KUBE_TEST_REPO_LIST and custom registries
K8s 1.13 organized and allowed overrides for where to pull
e2e images from. The env var needed is KUBE_TEST_REPO_LIST.

To facilitate users providing this, the following changes
have been made:

 - the user can specify a file path on the command line with
the proper contents
 - the file is read
 - the contents are used to create a configmap of the values
 - the configmap is mounted as a volume for the kube-conformance
container
 - the env var KUBE_TEST_REPO_LIST is set to point to this file

Fixes heptio#160

Signed-off-by: John Schnake <jschnake@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.