Registry mirrors authentication missing

[messiahUA]

/kind feature

1. Describe IN DETAIL the feature/behavior/change you would like to see.

There is a way to set registry mirrors which is great: https://github.com/kubernetes/kops/blob/master/docs/cluster_spec.md#registry-mirrors

But looks like there is no way to provide authentication or maybe I'm missing something.

2. Feel free to provide a design supporting your feature request.

Based on https://github.com/containerd/containerd/blob/main/docs/cri/registry.md#configure-registry-credentials config structure could look like:

spec:
  containerd:
    registryMirrors:
      docker.io:
      - endpoint: https://registry-1.docker.io
        username: ""
        password: ""
        auth: ""
        identitytoken: ""

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ddelange]

@hakman is there a way to achieve containerd auth with kOps?

Is the current option to use containerd ConfigOverride? i.e. I ssh into a node, copy /etc/containerd/config-kops.toml, modify it and pass it manually?

Similarly, I also didn't find a way to configure an insecure (http) registryMirror for containerd ref containerd/containerd#3847 (comment)

[ddelange]

/remove-lifecycle stale

[ddelange]

It seems containerd deprecated the plugins."io.containerd.grpc.v1.cri".registry.configs.YOUR_REGISTRY_DNS.auth method of supplying basicauth credentials for a registry ref
https://github.com/containerd/containerd/blob/22e8fc1f406dcca58122b23b73465939d3e99ba7/docs/cri/registry.md#configure-registry-credentials

[hakman]

Yes, we are planning to review this config in the near future.

[ddelange]

so we're currently using configOverride to manually add our private registry+basicauth to containerd like in my last two comments, which works fine.

but now we want to add GPU nodes to our cluster and enable the nvidia runtime for containerd (containerd 1.6.1 from kops 1.23 with k8s 1.23.5) ref https://github.com/kubernetes/kops/blob/v1.23.2/docs/gpu.md

which won't get injected due to the early return from configOverride ref https://github.com/kubernetes/kops/blob/v1.23.2/nodeup/pkg/model/containerd.go#L479-L503

@hakman do you have any suggestions how to move forward here in the meantime? is currently the only way to additionally manually inject the nvidia runtime in configOverride?

[hakman]

My preference is to add something similar to configOverride that only adds extra stuff to the generated (configAdditions or configExtras or ...). This way, one would be able to just add some desired extra config.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[hakman]

/remove-lifecycle stale
/lifecycle frozen

[ddelange]

hi @hakman 👋

should we get some extra thoughts on your last message?

for the full disclosure, I'm pasting our custom configOverride here, with the nvidia runtime and our private registry manually injected into a config (pulled from an older k8s v1.22 node):

version = 2

[plugins]

  [plugins."io.containerd.grpc.v1.cri"]

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
          runtime_type = "io.containerd.runc.v1"
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
            SystemdCgroup = true
            BinaryName = "/usr/bin/nvidia-container-runtime"

    # added for registry basicauth ref https://github.com/kubernetes/kops/issues/12916#issuecomment-1083066051
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."${local.docker_registry__dns}"]
          endpoint = ["${local.docker_registry__dns}"]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."${local.docker_registry__dns}".auth]
          # base64 encoded string from the concatenation of user:pass https://github.com/containerd/cri/pull/838/files#diff-a3d824da3c42420cd5cbb0a4a2c0e7b5bfddd819652788a0596d195dc6e31fa5R77
          auth = "${base64encode(var.docker_registry_basicauth)}"

[ddelange]

my hope is that a mechanism for this will still make it into kOps 1.26 but maybe that ship has sailed? ☸️

[hakman]

#16067 is adding configAdditions. Hopefully that is flexible enough for this use case for now.