Skip to content

Fix containerd 2.x registry TLS config via hosts.toml#5972

Merged
nvanthao merged 3 commits intomainfrom
fix/containerd-2x-registry-tls-hosts-toml
Apr 21, 2026
Merged

Fix containerd 2.x registry TLS config via hosts.toml#5972
nvanthao merged 3 commits intomainfrom
fix/containerd-2x-registry-tls-hosts-toml

Conversation

@nvanthao
Copy link
Copy Markdown
Member

Problem

Amazon Linux 2023 ships containerd 2.x by default. containerd 2.0 removed the plugins."io.containerd.grpc.v1.cri".registry.configs inline TOML approach from config.toml, breaking registry TLS configuration.

Solution

Migrated to the hosts.toml per-registry approach under /etc/containerd/certs.d/<server>/. This is the official containerd 2.x registry configuration method and aligns with containerd's current architecture.

Changes

  • scripts/distro/kubeadm/distro.sh — Rewrote kubeadm_registry_containerd_configure() to write per-registry hosts.toml files instead of appending inline TOML to config.toml
  • addons/containerd/template/base/install.sh — Added sed to set config_path = "/etc/containerd/certs.d" for containerd 1.x forward compatibility
  • .gitignore — Added AI/editor metadata folders and local workflow directories

Fixes #5161

Fixes #5161. containerd 2.0 removed inline registry.configs from config.toml.
Migrated to per-registry hosts.toml under /etc/containerd/certs.d/ to support
AL2023 (ships containerd 2.x by default) while maintaining 1.x compatibility.

- kubeadm: rewrite kubeadm_registry_containerd_configure() to use hosts.toml
- containerd addon: enable config_path in base install for future 1.x support
- gitignore: ignore AI/editor metadata and local workflow directories
@nvanthao nvanthao requested a review from a team as a code owner April 14, 2026 23:56
@nvanthao nvanthao added the type::feature An enhancement to an existing add on or feature label Apr 15, 2026
emosbaugh
emosbaugh previously approved these changes Apr 15, 2026
@nvanthao
Copy link
Copy Markdown
Member Author

I'm still testing the fix in Amazon Linux 2023 and other distribution.

containerd v2.x `config default` generates config_path with colon-separated
/etc/docker/certs.d suffix. The transfer service io.containerd.transfer.v1.local
silently ignores such paths and never reads hosts.toml, causing registry TLS
x509 verification failures. Strip the suffix to ensure hosts.toml is read.

Fixes: containerd/containerd#12415
@nvanthao
Copy link
Copy Markdown
Member Author

Sed ordering, for a fresh containerd v2.x default config:

  1. Step 1 (double-quote empty) - no-op, v2.x uses single quotes
  2. Step 2 (single-quote empty) - fixes transfer plugin '' → /etc/containerd/certs.d
  3. Step 3 (colon strip) - fixes CRI images plugin colon-separated → /etc/containerd/certs.d

Tested on Amazon Linux 2023 / containerd 2.2.1 / kURL kubeadm-based cluster against the latest commits.

What we verified

Config state after install.sh runs:

Plugin Before After
io.containerd.cri.v1.images.registry.config_path '/etc/containerd/certs.d:/etc/docker/certs.d' '/etc/containerd/certs.d'
io.containerd.transfer.v1.local.config_path '' '/etc/containerd/certs.d'

Both gaps we originally identified are now addressed:

  • The single-quote variant (config_path = '') is handled by the new sed - this is what containerd v2.x generates on Amazon Linux 2023.
  • The transfer plugin config_path is now set - required for any ctr-based operations (push/pull) that bypass the CRI images plugin.

End-to-end pull test passed:

  • crictl pull docker.io/library/busybox:latest
  • Pushed busybox:latest (linux/amd64) to kURL registry at 10.96.x.x:443
  • Pod in namespace kurl referencing 10.96.x.x/busybox:latest reached 1/1 Running - no x509 errors ✅

Idempotent on re-run - none of the three seds match after the first apply.

@nvanthao
Copy link
Copy Markdown
Member Author

Tested on Ubuntu 22.04 / containerd v1.7.29

Verified the containerd 1.x config and end-to-end registry flow on a kURL single-node cluster.

/etc/containerd/config.toml checks

  • config_path = "/etc/containerd/certs.d" present under [plugins."io.containerd.grpc.v1.cri".registry]
  • config_path = "/etc/containerd/certs.d" present under [plugins."io.containerd.transfer.v1.local"] (no colon-separated suffix)
  • SystemdCgroup = true still intact under runc.options
  • No legacy inline registry.configs TLS block remaining in the file

Per-registry hosts.toml checks

  • /etc/containerd/certs.d/10.96.3.39/hosts.toml created by kubeadm_registry_containerd_configure
  • server = "https://10.96.3.39" and ca = ["/etc/kubernetes/pki/ca.crt"] correctly set
  • Idempotency confirmed — re-running the configure function skips if hosts.toml already exists

End-to-end registry flow

  • crictl pull docker.io/library/busybox:latest pulls linux/amd64 image successfully
  • ctr images push pushes busybox to 10.96.3.39/kurl/busybox:latest (kurl namespace) with kube CA cert — no TLS errors
  • Pod with imagePullPolicy: Always using 10.96.3.39/kurl/busybox:latest scheduled, image pulled in ~109ms, container started and ran successfully

@nvanthao nvanthao requested a review from emosbaugh April 20, 2026 23:31
@nvanthao nvanthao merged commit d3de62b into main Apr 21, 2026
8 checks passed
@nvanthao nvanthao deleted the fix/containerd-2x-registry-tls-hosts-toml branch April 21, 2026 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type::feature An enhancement to an existing add on or feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

registry addon not work with containerd

3 participants