Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better UX for using local registry #1213

Open
h7kanna opened this issue Dec 30, 2019 · 24 comments
Open

Better UX for using local registry #1213

h7kanna opened this issue Dec 30, 2019 · 24 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@h7kanna
Copy link

h7kanna commented Dec 30, 2019

What would you like to be added:

Add a special node to the Kind cluster with 'registry' role. This node can run docker registry without the user having to run any extra scripts to configure Kind cluster

Why is this needed:

This can be an alternative to 'kind load docker-image' and naturally integrates with 'docker push' CI workflows.

Also, the registry storage can be mounted from the host and kind load can be avoided for easier setup.

@h7kanna h7kanna added the kind/feature Categorizes issue or PR as related to a new feature. label Dec 30, 2019
@BenTheElder
Copy link
Member

BenTheElder commented Jan 2, 2020

We're still designing this, there are some tradeoffs that depend on how the networking changes to fix #148

The obvious "registry" role route has some downsides, including that you'll need to push to each cluster.

@BenTheElder BenTheElder added the kind/design Categorizes issue or PR as related to design. label Jan 2, 2020
@BenTheElder BenTheElder self-assigned this Jan 7, 2020
@diafour
Copy link

diafour commented Jan 14, 2020

I choose not to run a local registry in-cluster. It is not great for testing against multiple versions and/or clusters.
So I run local registry as a standalone docker container with mounted volume in a home directory. Kind cluster is started with a small script, that does two things after the cluster is ready: it adds localhost:5000 as a trusted registry to /etc/containerd/config.toml and creates a NodePort Service and a DaemonSet to proxying from that NodePort to docker host IP.

That is an adopted solution back from kubeadm-dind-cluster days: kubernetes-retired/kubeadm-dind-cluster#56 (comment)

@BenTheElder
Copy link
Member

thanks for the input!

... it adds localhost:5000 as a trusted registry to /etc/containerd/config.toml ...

you can do this at boot time now, xref: https://kind.sigs.k8s.io/docs/user/local-registry/

... and creates a NodePort Service and a DaemonSet to proxying from that NodePort to docker host IP. ...

That's a neat approach, thanks :-)

@diafour
Copy link

diafour commented Jan 14, 2020

Yeah, that config.toml override was for 0.5 release. Thanks for the link!

@diafour
Copy link

diafour commented Jan 14, 2020

Hooray! I've just realized that endpoint should not be the same as mirror name and simplified my setup: no proxying with DaemonSet required, just containerdConfigPatches:
https://gist.github.com/diafour/13cef191b7cf39543393d310dd6353a0

The workflow is like this:


docker build -t "localhost:5000/name:tag" .
docker push localhost:5000/name:tag

kind: Pod
...
    image: localhost:5000/name:tag

cluster-15.sh create
kubectl replace ... 

There are create and delete commands to address #148 ;)

@BenTheElder Thanks again!

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 13, 2020
@BenTheElder BenTheElder added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 14, 2020
@kubernetes-sigs kubernetes-sigs deleted a comment from fejta-bot Apr 14, 2020
@nicks
Copy link
Contributor

nicks commented Apr 14, 2020

Hi! I work on Tilt! We care so much about good local registry support that if you're not using a local registry, we emit a warning that you should be.

We currently point people at this script to set it up:
https://github.com/windmilleng/kind-local/blob/master/kind-with-registry.sh
but we'd love to recommend a solution built into Kind.

I think the main things that are important to us (from a UX perspective) are (in order):

  1. The ability for tooling to auto-detect the registry from the cluster
  2. The ability to delete the cluster and create a new one without deleting the registry

So I think our ideal CLI UX would be something like:

kind create registry [--name optional-name]
kind create cluster --with-registry default

where --with-registry default annotated the Kubernetes nodes with the URL of the registry (as the kind-with-registry.sh script above does)

@BenTheElder BenTheElder added this to the 2020 goals milestone Apr 25, 2020
@BenTheElder
Copy link
Member

still on my radar! we've mostly landed the host reboot work which includes using a docker network w/ embeded DNS, but had to spend time on a number of other critical fixes.

we still need some follow up to that, but once we settle down the new networking a bit we should have a better idea what we can build the registry integration against.

@BenTheElder BenTheElder added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label Apr 26, 2020
@BenTheElder
Copy link
Member

I think instead of a "create registry" w/ ordering issues (and presumably failing if it already exists etc.) the ux from k3d is closer to what people need, ie. --enable-registry when creating the cluster. (which enables a shared local registry). https://github.com/rancher/k3d/blob/master/docs/registries.md#using-the-k3d-registry

@BenTheElder
Copy link
Member

(and I've just filed what I hope is the last major fix related to the DNS stuff, we neglected to update the NO_PROXY logic to handle it properly)

@Morgma
Copy link

Morgma commented Apr 22, 2021

Hooray! I've just realized that endpoint should not be the same as mirror name and simplified my setup: no proxying with DaemonSet required, just containerdConfigPatches:
https://gist.github.com/diafour/13cef191b7cf39543393d310dd6353a0

The workflow is like this:


docker build -t "localhost:5000/name:tag" .
docker push localhost:5000/name:tag

kind: Pod
...
    image: localhost:5000/name:tag

cluster-15.sh create
kubectl replace ... 

There are create and delete commands to address #148 ;)

@BenTheElder Thanks again!

@diafour, I'm using Docker for Windows running on WSL 2, Ubuntu 18.04. When I run kind-with-registry.sh, I can push/pull from my host at localhost:5000, however, k8s fails to pull images from the registry from that endpoint. I've tried substituting "host.docker.internal", but container pulls still fail to resolve the endpoint. Am I missing some specific guidance for Docker on WSL2?

Thanks!

EDIT:

Somewhere along the line either I wasn't running 'docker network connect...' or I wasn't annotating the node with "kind.x-k8s.io/registry=localhost:5000". My image pulls are resolving now.

@MadhavJivrajani
Copy link

/remove-kind design
/kind feature
kind/design is migrated to kind/feature, see kubernetes/community#6144 for more details

@k8s-ci-robot k8s-ci-robot removed the kind/design Categorizes issue or PR as related to design. label Oct 11, 2021
@BenTheElder BenTheElder removed this from the v0.11.0 milestone Oct 14, 2021
@BenTheElder BenTheElder removed their assignment Oct 14, 2021
@asilverman
Copy link

Consider using a different port than 5000 since its used by Apple MacOS Monterrey

@BenTheElder
Copy link
Member

BenTheElder commented Dec 30, 2021

That is wildly unfortunate, 5000 is a commonly used development port and docker special cases allowing localhost:5000 as an HTTP (not HTTPS) endpoint out of the box.

Reconfiguring dockerd is not something kind ought to be doing and will introduce user friction.

Not excited to make the experience worse on other platforms because Apple decided to suddenly start binding to a commonly used port OOTB :/

@diafour
Copy link

diafour commented Jan 8, 2022

Just for the record, there is no special case localhost:5000. dockerd trusts all local IPs despite the port:

Local registries, whose IP address falls in the 127.0.0.0/8 range, are automatically marked as insecure as of Docker 1.3.2. It is not recommended to rely on this, as it may change in the future.

Insecure registries

@BenTheElder
Copy link
Member

Well I stand corrected 🙃 , thank you for sharing @diafour 🙏

If someone wants to update the port in the sample script, that should be a trivial change then 🙃

@HummingMind
Copy link

Hello!
Is this still being worked on? If so, any ETA? I see it was put back into the backlog. :(

Thank you!

@BenTheElder
Copy link
Member

I’m sorry but we’re prioritizing bug fixes like fixing reboot support and keeping up with Kubernetes, and user support, it doesn’t leave much time for designing / reviewing / implementing new features at the moment.

@HummingMind
Copy link

I’m sorry but we’re prioritizing bug fixes like fixing reboot support and keeping up with Kubernetes, and user support, it doesn’t leave much time for designing / reviewing / implementing new features at the moment.

That is fair. Hopefully that changes in the near future.
Thank you for the work you guys do on KIND!

@BenTheElder
Copy link
Member

Thanks, the core maintainers work primarily on Kubernetes but we're trying 😅

Smaller updates:

@BenTheElder BenTheElder mentioned this issue Oct 25, 2022
@colin-williams
Copy link

colin-williams commented Dec 14, 2022

With or without using a local registry, would it be possible to create a configuration setting to "proxy" container images automagically through the local docker host?

That way, bringing up or taking down kind clusters we'd still have the images locally. This would mitigate useless downloads that waste bandwidth, time, and energy. As a bonus it would be nice if the image didn't have to be duplicated between kind and the local docker store.

@BenTheElder
Copy link
Member

With or without using a local registry, would it be possible to create a configuration setting to "proxy" container images automagically through the local docker host?

Long story short: not readily, but it should be technically feasible if someone wished to invest in it.

Currently the KIND maintainers are pretty focused on bug-fixes and maintenance when we're working on KIND (and not say, Kubernetes core).

Containerd does have a way to configure * host to a mirror, except they changed the mirroring config (#2875, #2952) in a way that may not permit that anymore?, and you'd need a registry that implemented a recent-ish OCI distribution header (containerd/containerd#3734) to detect the real target host in the pull and mirror all hosts. I'm not aware of such an implementation yet, previously this wasn't feasible because the server didn't know what original host the target image was being mirror-pulled from.

As a bonus it would be nice if the image didn't have to be duplicated between kind and the local docker store.

Docker does not document or expose the local docker image storage except through pull/push/save/load, all of which do not match registries in that they are single-architecture (IE no manifest lists, which docker can do, but only through a special subcommand with different storage, awkwardly). We also support podman, and nominally support remote dockerd.

@nicks
Copy link
Contributor

nicks commented Dec 14, 2022

FWIW, there's work actively under way to switch Docker's image store to use containerd, though it's still in beta - https://www.docker.com/blog/extending-docker-integration-with-containerd/

Docker has talked a little bit about how we might make it possible to auto-magically make the images appear in Kind. As Ben said, it's theoretically possible but there are a lot of devils hiding in the details. Might have to try a few different approaches.

@BenTheElder
Copy link
Member

FWIW, there's work actively under way to switch Docker's image store to use containerd, though it's still in beta - https://www.docker.com/blog/extending-docker-integration-with-containerd/

That's neat but it still won't be the node level containerd, most of that metadata is in a boltdb that you can't share between instances. (Also we can't just read files from the kind binary POV since they may be on different machines) not to mention podman which has its own storage.

The only approach I see here as semi-feasible is if docker and podman etc (see also requests for nerdctl) get a special image pull proxy that reads the above header and then checks local storage but that's kinda fraught with nasty details (like how do you force pulling a new version of a tag)

If you're willing to pay the cost and just want to cache between runs without knowing the images and explicitly storing / loading / mirroring them, you can use an extraMount and mount the containerd storage through to a directory on the host. However this will grow unbounded as for other reasons we have no image GC enabled in kind (which has it's own pile of issues that I won't dive into here)

@Nossnevs
Copy link

Nossnevs commented Feb 2, 2023

One thing that should at least be added to the page https://kind.sigs.k8s.io/docs/user/local-registry/ is how to handle proxy setting.

If you have set the http_proxy environment variable, you have to add kind-registry to no_proxy variable. If you don't then it will try to use the proxy to find it. Not the easiest to find because I for example got ErrImagePull forbidden in Kubernetes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Development

No branches or pull requests