-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
imagePullSecrets support #502
Comments
I wouldn't even call image preloading a workaround. I had a deployment that builds a custom Caddy server, packaged as a Docker image with a static website and additional configuration. I push it to my private GitLab registry, and redeploy it on any code changes. I could preload an initial image, but I don't know how I'd push new changes into the cluster later. If this isn't supported, I'm hoping there's another solution. |
@ndarilek I second that, we need private registries for most cases, hopefully this issue will go further. Thanks, Simone. |
There may be some underlying issue with containerd/cri pull secrets support where it only works with gcr registry containerd/cri/issues/668, it may be possible we need to update our version of cri, or could be just a config issue. It may help to try creating a file |
@erikwilson Sure I'll give it a try and let you know if it works with the additional configuration, many thanks! |
imagePullSecrets works for me with Docker runtime (using --docker option). |
Hmm, it occurred to me that this might be an issue with the container
subsystem. Has anyone tried using Docker? Does the Docker subsystem
support this?
I don't mind using Docker instead of Containerd, but I do need this
feature. Is that something I can do after I've already launched my
agent? I'm just using local paths for storage, so in theory everything
would just re-instantiate with Docker and not Containerd, but...
|
Great news. Can I just add the `--docker` option to a systemd unit
override and restart k3s to switch engines? Or am I out of luck now that
I'm using containerd?
Thanks.
|
I appended `--docker` to the command line, and things seem to still be
working. I still have /var/lib/rancher/k3s/agent/containerd, though.
Would that be true even using the Docker engine? The string "docker"
doesn't appear anywhere in journalctl.
|
I learned that `kubectl get nodes -o wide` outputs the container runtime
being used. But after adding the `--docker` argument, I get:
```
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP
OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-000 Ready <none> 2d16h v1.14.1-k3s.4 195.201.35.232 <none>
Fedora 29 (Twenty Nine) 5.0.17-200.fc29.x86_64
containerd://1.2.5+unknown
```
So it still claims containerd. Yet I did submit a pod with:
```
imagePullSecrets:
- name: website-registry
```
in its specification and it was deployed.
Full disclosure: I didn't even try `imagePullSecrets` without the
--docker argument since this issue suggested it wouldn't work. So either
it does, or `kubectl get nodes -o wide` is misrepresenting the container
runtime and I should file a separate issue for that.
|
@simmessa were you able to get this working? |
Worked for me with private registry in dockerhub. 👍 |
I tried this with a private gitlab registry and it didn't work. Here is my [plugins.opt]
path = "/var/lib/rancher/k3s/agent/containerd"
[plugins.cri]
stream_server_address = "ablanco"
stream_server_port = "10010"
[plugins.cri.cni]
bin_dir = "/var/lib/rancher/k3s/data/4e1224c66a9dbb9b03daefff200f4f8eaf45590fb722b6fe2924a201d6de2e8d/bin"
conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"
[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."gitlab.private.com"]
endpoint = ["http://gitlab.private.com:5005"]
[plugins.cri.registry.mirrors."gitlab.private.com:5005"]
endpoint = ["http://gitlab.private.com:5005"]
[plugins.cri.registry.auths]
[plugins.cri.registry.auths."gitlab.private.com"]
username = "myuser"
password = "mypassword"
# also tried with the auth token from docker config with no luck
#auth = "csisddddd5jbzpzcGFuVFIxMg==" When I try to execute: k3s crictl --debug pull gitlab.private.com:5005/mygroup/myimage:latest I get: DEBU[2019-06-10T16:53:30.803820935-03:00] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gitlab.private.com:5005/mygroup/myimage:latest,},Auth:nil,SandboxConfig:nil,}
DEBU[2019-06-10T16:53:30.856669975-03:00] PullImageResponse: nil
FATA[2019-06-10T16:53:30.856716915-03:00] pulling image failed: rpc error: code = Unknown desc = failed to resolve image "gitlab.private.com:5005/mygroup/myimage:latest": no available registry endpoint: failed to fetch anonymous token: unexpected status: 403 Forbidden |
Nope, I made a try with the containerd toml template but I might have missed something and it I wasn't able to make it work. I might make another try though. |
Works for me (containerd backend) without any changes using a private gcr.io registry. |
I had added below to config.toml to enable private registery from dockerhub.
|
Did it Worked for you? I am also struggling hard to make it work. Tried with first with tls also. Please let me know if worked for you. |
For those having an issue with the above snippets : LIke this:
Side note - I have not dug into the proper grc.io auth setup, so just using |
I am super confused. The documented config in containerd specifies a completly different config layout. I tried out all combinations. The code snippet by @seerickcode seems to work, but only as far to make the creds show up in If I were to execute https://github.com/containerd/cri/blob/master/docs/registry.md
|
@upMKuhn Double check what you have typed. It is not the same as what I have typed :) edit: tried killing the line, restarting k3s, and checking my I am using k3s version v0.8.1 (d116e74) Also the Auth:nil shows up even in a working config. FYI, I have, since the last posting, switched to a using a service account so I have a persistent token. For those trying that method, this is what it looks like (in
|
Just in case somehow username/password based auth had somehow changed since I last posted (not sure what k3s version I had at the time), I went ahead and just added another repo that was not on my k3s. I used an actual gitlab based repo too, just to make sure we're on the same page. As a bonus, just to add some more working examples (well working for me and redacted), this configuration is for an url with a non-standard port, so you can see an example of how that is setup. This is also the FULL Pulling things down just fine with example like : k3s also pulling fine as the containers are standing up.
|
Awesome thank you very much @seerickcode . For anyone facing the same issue: Does anyone know if I will need to specify this config on all my future nodes or is it only required on the master? |
well, in my case i deployed k3s 0.10.1 (latest release), one master and one node, with default options (containerd, not docker) i deployed without problems private containers from registry.gitlab.com, so works with me in this case, created the token in gitlab.com settings, passing this as a secret to k3s and kubectl apply private_deployment,yaml with imagePullSecrets deploy the pods without problems |
Same .... I've just set up an image pull secret on my cluster. (Latest version I think) It pulled the image from my private repo without any problems. So I think this can be closed? |
Not knowing this was an issue, I was able to deploy a secret for my git lab repository and pull images. Then I deleted my deployment and re-deployed and was getting ErrImagePull errors. Images are present using crictl images, so this did work at one point, but is failing now. Un-installed and now it is failing to pull on the first pass. k3s version v1.19.2+k3s1 (d38505b) (on WSL2 - windows server 2004, but doesn't seem to be related) |
I'm not aware of any issue with imagePullSecrets. Can you share a sample deployment that's giving you trouble? From reading the thread at containerd/containerd#3291 it appears that some registries don't implement the registry API properly - they use nonstandard URLs or return unexpected status codes in the authentication workflow. Have you tried this against a different registry? |
Using this with private gitlab.com repository and no docker. I'm experiencing no issues as of today. |
when you Install k3s . Use docker instead of containerd,Don‘t use embedded crictl。 Install command: my k3s version v1.18.9+k3s1 (630bebf) |
It works for me only when adding a docker secret with username/password/email, not with a generic secret containing the Docker config file with the key. |
Closing due to age. If anyone is experiencing a problem with this please open a new issue. Most cases it's incorrect configuration or a misunderstanding about how the containerd registry configuration works. |
I'm confused. Why was this closed? The original post seems like a pretty clear request for k3s to implement I also see #3314 suggesting |
Image pull secrets are a core Kubernetes feature. They don't need to be implemented or require any additional system configuration. As I said above, if you're having a problem using them as documented upstream then please open an issue. |
The use of imagePullSecrets should be supported in the same way as in Kubernetes, but it seems this is not the case, this is not only limited to initial setup as suggested by #99 but should be available even after booting up the cluster.
Is your feature request related to a problem? Please describe.
I'm not 100% sure this is a bug, but I'm pretty convinced anybody using k8s expects to be able to use imagePullSecrets for deployments and this is not possible at this point. Since this is not officially communicated as a feature I will say it's a feature request, but for a pretty basic feature, from my point of view.
Describe the solution you'd like
I'd like to be able to use imagePullSecrets in deployments for pulling container images from private registries
Describe alternatives you've considered
I know there's another way to achieve this, via document image preloading, but it's clearly a workaround and I believe this kind of 'basic functionality' should be made available as part of a larger k8s standard
Additional context
Here's a link to the situation described pretty well on stackoverflow: https://stackoverflow.com/questions/55457376/imagepullsecrets-not-working-when-using-k3s
The text was updated successfully, but these errors were encountered: