[ENHANCEMENT] Ease using a private registry

[iwilltry42]

Currently, enable pulling from a private registry from withing k3d can be achieved e.g. by following this example: https://rancher-users.slack.com/archives/CHM1EB3A7/p1561743458058500 posted by Nicolas Levée in our Slack channel.
Here's a copy of that post:

I don't know if this can help but for private registries (like GCR), here is my Solution :
I start k3d with this create command :
k3d create --volumes ${K3D_BASE_PATH}/config.toml.tmpl:/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
The content of config.toml.tmpl :

path = "{{ .NodeConfig.Containerd.Opt }}"
[plugins.cri]
stream_server_address = "{{ .NodeConfig.AgentConfig.NodeName }}"
stream_server_port = "10010"
{{- if .IsRunningInUserNS }}
disable_cgroup = true
disable_apparmor = true
restrict_oom_score_adj = true
{{ end -}}
{{- if .NodeConfig.AgentConfig.PauseImage }}
sandbox_image = "{{ .NodeConfig.AgentConfig.PauseImage }}"
{{ end -}}
{{- if not .NodeConfig.NoFlannel }}
  [plugins.cri.cni]
    bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
    conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"
{{ end -}}

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."docker.io"]
    endpoint = ["https://registry-1.docker.io"]
  [plugins.cri.registry.mirrors."eu.gcr.io"]
    endpoint = ["https://eu.gcr.io"]
[plugins.cri.registry.auths]
  [plugins.cri.registry.auths."https://eu.gcr.io"]
    username = "oauth2accesstoken"
    password = "....."

The first part is from k3s source code (https://github.com/rancher/k3s/blob/master/pkg/agent/templates/templates.go)The second part add some registries in configMy images from eu.gcr.io/project/image_name can be pull without any problem

I'd like to ease the use of this, e.g. by external configuration via environment variables, a config file, or via additional flags for the create command.

[matthiasak]

+1 for this

[d11wtq]

Yeah I want to use k3d as an alternative to minikube, where you can docker build an image within minikube (via minikube docker-env) and Kubernetes can run that image because it's already present. Doesn't seem to be a way to get this working with k3d and Docker for Mac right now, aside from running a registry as detailed above? I tried using --server-arg --docker and mounting /var/run/docker.sock, but k3d then fails to create pods with Failed create pod sandbox: rpc error: code = Unknown desc = rewrite resolv.conf failed for pod ....

[iwilltry42]

@d11wtq the k3s image that we use, doesn't work with the --docker flag, so that won't work unfortunately.
Instead of running a registry, you can use the import-images command to import pre-built images from your docker daemon into k3d.
Unfortunately, docker is using a different image storage format than containerd, so we cannot simply mount the image cache into k3d.

[iwilltry42]

The very best news for us would be the success of this: moby/moby#38043
Making docker to use the containerd storage backend 👍
We could as well try to include something like BuildKit in k3d, which we could use for building inside k3d using e.g. k3d build image. Would require a shared caching/storage layer though to avoid having to build large images in each k3d node.

[tux-00]

This feature is a must have!

[iwilltry42]

Closed via #161 and follow-up changes