Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Insecure Registry Support #145

Open
nnordrum opened this issue Mar 4, 2019 · 24 comments

Comments

@nnordrum
Copy link

commented Mar 4, 2019

Is your feature request related to a problem? Please describe.
As a user, I cannot access insecure registries from my k3s instance.

Describe the solution you'd like
Add --insecure-registry to server and/or node.

Describe alternatives you've considered
I'm specifically trying to address this with the proxy team, but it should be added regardless.

Additional context
Behind a corporate proxy, it might use a different cert. It is what it is...

@ibuildthecloud

This comment has been minimized.

Copy link
Member

commented Mar 6, 2019

I think it make sense to add --insecure-registry argument to the agent to address this.

@ibuildthecloud ibuildthecloud added this to Backlog in K3S Development via automation Mar 6, 2019

@xiaosuiba

This comment has been minimized.

Copy link

commented Mar 6, 2019

@ibuildthecloud I think expose a --containerd-config argument also make sense. With the configuration file, I could customize containerd with insecure-registry, sandbox-image, etc.

@nnordrum

This comment has been minimized.

Copy link
Author

commented Mar 13, 2019

+1 on the --containerd-config

@debianmaster

This comment has been minimized.

Copy link

commented Apr 29, 2019

+1 for --containerd-config

@AlbertoPeon

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

👍 this would be nice to have !

Are there any alternatives to make it work with an insecure registry at the moment? Or setting a custom CA somewhere? Thanks!

@oraoto

This comment has been minimized.

Copy link

commented May 7, 2019

In k3s 0.5, you can config containerd with the template file <data-dir>/etc/containerd/config.toml.tmpl.

To add insecure registory:

[plugins.opt]
path = "{{ .NodeConfig.Containerd.Opt }}"

[plugins.cri]
stream_server_address = "{{ .NodeConfig.AgentConfig.NodeName }}"
stream_server_port = "10010"

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."docker.io"]
    endpoint = ["https://registry-1.docker.io"]
  [plugins.cri.registry.mirrors."your-insecure-registory:5000"]
    endpoint = ["http://your-insecure-registory:5000"]

Doc: containerd and Docker

@debianmaster

This comment has been minimized.

Copy link

commented May 7, 2019

@oraoto do you know how to configure for selfsigned registries?

@oraoto

This comment has been minimized.

@flxs

This comment has been minimized.

Copy link

commented May 9, 2019

@debianmaster I've been playing around with this while waiting for the proper fix, and I was able to get it to work by adding the CA cert to the host system's trusted CAs (Ubuntu 18.04 in my case) and configuring it via kubectl like any other registry.

@debianmaster

This comment has been minimized.

Copy link

commented May 10, 2019

@flxs can you expand this a bit more? what have you done? configuring it via kubectl like any other registry. . is it adding registry secret and linking to service account?

@SwagMuffinMcYoloPants

This comment has been minimized.

Copy link

commented May 14, 2019

In k3s 0.5, you can config containerd with the template file <data-dir>/etc/containerd/config.toml.tmpl.

To add insecure registory:

[plugins.opt]
path = "{{ .NodeConfig.Containerd.Opt }}"

[plugins.cri]
stream_server_address = "{{ .NodeConfig.AgentConfig.NodeName }}"
stream_server_port = "10010"

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."docker.io"]
    endpoint = ["https://registry-1.docker.io"]
  [plugins.cri.registry.mirrors."your-insecure-registory:5000"]
    endpoint = ["http://your-insecure-registory:5000"]

Doc: containerd and Docker

I couldn't get this to work until I changed

[plugins.cri]
stream_server_address = "{{ .NodeConfig.AgentConfig.NodeName }}"
stream_server_port = "10010"

to

[plugins.cri]
stream_server_address = "{{ .NodeConfig.AgentConfig.NodeName }}"
stream_server_port = "10010"
  [plugins.cri.cni]
    bin_dir = "/bin"
    conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"
@rcarmo

This comment has been minimized.

Copy link

commented May 18, 2019

Actually, that should be:

[plugins.cri]
stream_server_address = "{{ .NodeConfig.AgentConfig.NodeName }}"
stream_server_port = "10010"
  [plugins.cri.cni]
    bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
    conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"

At least that is what works for me with v0.5.0.

Here's the full config.toml.tmpl:

[plugins.opt]
path = "{{ .NodeConfig.Containerd.Opt }}"

[plugins.cri]
stream_server_address = "{{ .NodeConfig.AgentConfig.NodeName }}"
stream_server_port = "10010"
  [plugins.cri.cni]
    bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
    conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."docker.io"]
    endpoint = ["https://registry-1.docker.io"]
  [plugins.cri.registry.mirrors."registry.lan:5000"]
    endpoint = ["http://registry.lan:5000"]
@rcarmo

This comment has been minimized.

Copy link

commented Jul 20, 2019

FYI, this stopped working for me with 0.7.0 on armhf. I have yet to figure out why.

@Duske

This comment has been minimized.

Copy link

commented Jul 22, 2019

I cannot get it to work as well. I use 0.7.0 and default k3s

@Duske

This comment has been minimized.

Copy link

commented Jul 26, 2019

@rcarmo This issue could be related to it: containerd/cri#1201, especially the merged PR containerd/containerd#3400

@rcarmo

This comment has been minimized.

Copy link

commented Jul 28, 2019

Thanks. I’m going to try reverting to 0.6.0 in the meantime. since I really need support for an HTTP-only registry for dev/test/CI (it is a royal pain to do certificates for .lan/.local/anything private, and not worth the hassle).

@Duske

This comment has been minimized.

Copy link

commented Aug 1, 2019

The problem was a not 100%-conform HTTP API V2 custom Docker registry.
After fixing the registry, it works with k3s in version 0.7.0.

@rcarmo

This comment has been minimized.

Copy link

commented Aug 1, 2019

@keksecops

This comment has been minimized.

Copy link

commented Aug 2, 2019

Any news on this issue ? I'm having the same problem here I guess.
The output of crictl info regarding registries:

...
   "registry": {
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://registry-1.docker.io"
          ]
        },
        "xxx.yyy.zzz:5000": {
          "endpoint": [
            "http://xxx.yyy.zzz:5000"
          ]
        }
      },
      "auths": null
    },
...

And the error that it gives me :
pulling image failed: rpc error: code = Unknown desc = failed to resolve image "xxx.yyy.zzz:5000/abc/alpine-curl:latest": no available registry endpoint: failed to do request: Head https://xxx.yyy.zzz:5000/v2/abc/alpine-curl/manifests/latest: http: server gave HTTP response to HTTPS client

But I can pull the image fine on my machine using docker.
Any advice ?

@rcarmo

This comment has been minimized.

Copy link

commented Aug 2, 2019

@solsson

This comment has been minimized.

Copy link

commented Aug 13, 2019

The status of this issue wasn't entirely clear to me, but I can report that with current installer (d8c4f38), k3s v0.8.0 and multipass (Ubuntu 18.04.2) I get a working insecure registry support box after:

INSTALL_K3S_SKIP_START=true ./install.sh
mkdir -p     /var/lib/rancher/k3s/agent/etc/containerd
cat <<EOF >> /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
[plugins.opt]
path = "{{ .NodeConfig.Containerd.Opt }}"

[plugins.cri]
  stream_server_address = "{{ .NodeConfig.AgentConfig.NodeName }}"
  stream_server_port = "10010"

  [plugins.cri.cni]
    bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
    conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"

  [plugins.cri.registry]
    [plugins.cri.registry.mirrors]
      [plugins.cri.registry.mirrors."my.registry.local"]
        endpoint = ["http://my.registry.local"]
EOF
service k3s start
#  confirm using k3s crictl info

Found in a different issue that the template to base the changes on is in https://github.com/rancher/k3s/blob/master/pkg/agent/templates/templates.go. I guess it will evolve.

@rcarmo

This comment has been minimized.

Copy link

commented Aug 13, 2019

@solsson

This comment has been minimized.

Copy link

commented Aug 13, 2019

@rcarmo I didn't bother with adding that one as it appears in crictl info anyway, and it works:

    "registry": {
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://registry-1.docker.io"
          ]
        },
        "my.registry.local": {
          "endpoint": [
            "http://my.registry.local"
          ]
        }
      },
      "auths": null
    },
@rcarmo

This comment has been minimized.

Copy link

commented Aug 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.