Invalid hosts.toml for private registry is generated

[kurokobo]

Environmental Info:
K3s Version: v1.29.3+k3s1 (8aecc26)

k3s version v1.29.3+k3s1 (8aecc26b)
go version go1.21.8

Node(s) CPU architecture, OS, and Version: CentOS Stream 9

Linux <hostname> 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

Single node cluster

Describe the bug:

By adding following simple /etc/rancher/k3s/registries.yaml and restarting k3s, invalid hosts.toml for containerd is generated and causes configs to be ignored.

Steps To Reproduce:

• Install K3s: curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.29.3+k3s1 sh -s - --write-kubeconfig-mode 644
• Add following registries.yaml to specify credentials and to skip certificate verification

  sudo tee /etc/rancher/k3s/registries.yaml <<EOF
  configs:
    registry.example.com:
      auth:
        username: reguser
        password: Registry123!
      tls:
        insecure_skip_verify: true
  EOF

• Restart k3s: sudo systemctl restart k3s

Expected behavior:

Generated hosts.toml is correct and pulling from private registry is completed without any errors

Actual behavior:

• Generated hosts.toml causes error="invalid `host` tree" since there is no [host."https://registry.example.com/v2"]
• Pulling from private registry is failed since any configuration in hosts.toml is ignored

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/registry.example.com/hosts.toml
# File generated by k3s. DO NOT EDIT.

server = "https://registry.example.com/v2"
capabilities = ["pull", "resolve", "push"]

skip_verify = true

$ sudo $(which k3s) ctr images pull --hosts-dir "/var/lib/rancher/k3s/agent/etc/containerd/certs.d" --user 'reguser:Registry123!' registry.example.com/demo/demo:demo
ERRO[0000] failed to decode hosts.toml                   error="invalid `host` tree"
INFO[0000] trying next host                              error="failed to do request: Head \"https://registry.example.com/v2/demo/demo/manifests/demo\": tls: failed to verify certificate: x509: certificate signed by unknown authority" host=registry.example.com
ctr: failed to resolve reference "registry.example.com/demo/demo:demo": failed to do request: Head "https://registry.example.com/v2/demo/demo/manifests/demo": tls: failed to verify certificate: x509: certificate signed by unknown authority

Additional context / logs:

Adding mirrors does not help.

$ sudo tee /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  registry.example.com:
    endpoint:
      - https://registry.example.com
configs:
  registry.example.com:
    auth:
      username: reguser
      password: Registry123!
    tls:
      insecure_skip_verify: true
EOF

$ sudo systemctl restart k3s

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/registry.example.com/hosts.toml
# File generated by k3s. DO NOT EDIT.

server = "https://registry.example.com/v2"
capabilities = ["pull", "resolve", "push"]

skip_verify = true

Exact the same configration and steps work as expected on the older v1.28.7+k3s1.

$ k3s -v
k3s version v1.28.7+k3s1 (051b14b2)
go version go1.21.7

By v1.28.7+k3s1, the same registries.yaml will generate following hosts.toml that contains [host."https://registry.example.com/v2"].

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/registry.example.com/hosts.toml
# File generated by k3s. DO NOT EDIT.


[host."https://registry.example.com/v2"]
  capabilities = ["pull", "resolve"]
  skip_verify = true

Pulling from private registry with this hosts.toml works.

$ sudo $(which k3s) ctr images pull --hosts-dir "/var/lib/rancher/k3s/agent/etc/containerd/certs.d" --user 'reguser:Registry123!' registry.example.com/demo/demo:demo
registry.example.com/demo/demo:demo:                                              resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:f7394791f5b6030de1952e40803687fbcd307291591440f70ec781e33340af34: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:add061545bb927986a16eadb5de08c2b77262c58ee26db302ed5620e78534023:    done           |++++++++++++++++++++++++++++++++++++++| 
...
layer-sha256:9cd2056efb53587a8f5b28457317a34ce3388b9e6d15ba88c3783c2a5aeb93bc:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.1 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:f7394791f5b6030de1952e40803687fbcd307291591440f70ec781e33340af34...
done: 7.918237ms

I didn't dive into the code deeper yet, but since no [host."https://registry.example.com/v2"] is generated by v1.29.3+k3s1, I suspect the .Endpoints is empty by some reason.

• HostsTomlTemplate:

  k3s/pkg/agent/templates/templates.go

  Lines 43 to 88 in 8aecc26

  	const HostsTomlTemplate = `
  	{{- /* */ -}}
  	# File generated by {{ .Program }}. DO NOT EDIT.
  	{{ with $e := .Default }}
  	{{- if $e.URL }}
  	server = "{{ $e.URL }}"
  	capabilities = ["pull", "resolve", "push"]
  	{{ end }}
  	{{- if $e.Config.TLS }}
  	{{- if $e.Config.TLS.CAFile }}
  	ca = [{{ printf "%q" $e.Config.TLS.CAFile }}]
  	{{- end }}
  	{{- if or $e.Config.TLS.CertFile $e.Config.TLS.KeyFile }}
  	client = [[{{ printf "%q" $e.Config.TLS.CertFile }}, {{ printf "%q" $e.Config.TLS.KeyFile }}]]
  	{{- end }}
  	{{- if $e.Config.TLS.InsecureSkipVerify }}
  	skip_verify = true
  	{{- end }}
  	{{ end }}
  	{{ end }}
  	{{ range $e := .Endpoints -}}
  	[host."{{ $e.URL }}"]
  	capabilities = ["pull", "resolve"]
  	{{- if $e.OverridePath }}
  	override_path = true
  	{{- end }}
  	{{- if $e.Config.TLS }}
  	{{- if $e.Config.TLS.CAFile }}
  	ca = [{{ printf "%q" $e.Config.TLS.CAFile }}]
  	{{- end }}
  	{{- if or $e.Config.TLS.CertFile $e.Config.TLS.KeyFile }}
  	client = [[{{ printf "%q" $e.Config.TLS.CertFile }}, {{ printf "%q" $e.Config.TLS.KeyFile }}]]
  	{{- end }}
  	{{- if $e.Config.TLS.InsecureSkipVerify }}
  	skip_verify = true
  	{{- end }}
  	{{ end }}
  	{{- if $e.Rewrites }}
  	[host."{{ $e.URL }}".rewrite]
  	{{- range $pattern, $replace := $e.Rewrites }}
  	"{{ $pattern }}" = "{{ $replace }}"
  	{{- end }}
  	{{ end }}
  	{{ end -}}
  	`

• getHostConfigs():

  k3s/pkg/agent/containerd/config.go

  Lines 71 to 154 in 8aecc26

  	func getHostConfigs(registry *registries.Registry, noDefaultEndpoint bool, mirrorAddr string) HostConfigs {
  	hosts := map[string]templates.HostConfig{}
  	// create config for default endpoints
  	for host, config := range registry.Configs {
  	if c, err := defaultHostConfig(host, mirrorAddr, config); err != nil {
  	logrus.Errorf("Failed to generate config for registry %s: %v", host, err)
  	} else {
  	if host == "*" {
  	host = "_default"
  	}
  	hosts[host] = *c
  	}
  	}
  	// create endpoints for mirrors
  	for host, mirror := range registry.Mirrors {
  	// create the default config, if it wasn't explicitly mentioned in the config section
  	config, ok := hosts[host]
  	if !ok {
  	if c, err := defaultHostConfig(host, mirrorAddr, configForHost(registry.Configs, host)); err != nil {
  	logrus.Errorf("Failed to generate config for registry %s: %v", host, err)
  	continue
  	} else {
  	if noDefaultEndpoint {
  	c.Default = nil
  	} else if host == "*" {
  	c.Default = &templates.RegistryEndpoint{URL: &url.URL{}}
  	}
  	config = *c
  	}
  	}
  	// track which endpoints we've already seen to avoid creating duplicates
  	seenEndpoint := map[string]bool{}
  	// TODO: rewrites are currently copied from the mirror settings into each endpoint.
  	// In the future, we should allow for per-endpoint rewrites, instead of expecting
  	// all mirrors to have the same structure. This will require changes to the registries.yaml
  	// structure, which is defined in rancher/wharfie.
  	for i, endpoint := range mirror.Endpoints {
  	registryName, url, override, err := normalizeEndpointAddress(endpoint, mirrorAddr)
  	if err != nil {
  	logrus.Warnf("Ignoring invalid endpoint URL %d=%s for %s: %v", i, endpoint, host, err)
  	} else if _, ok := seenEndpoint[url.String()]; ok {
  	logrus.Warnf("Skipping duplicate endpoint URL %d=%s for %s", i, endpoint, host)
  	} else {
  	seenEndpoint[url.String()] = true
  	var rewrites map[string]string
  	// Do not apply rewrites to the embedded registry endpoint
  	if url.Host != mirrorAddr {
  	rewrites = mirror.Rewrites
  	}
  	ep := templates.RegistryEndpoint{
  	Config: configForHost(registry.Configs, registryName),
  	Rewrites: rewrites,
  	OverridePath: override,
  	URL: url,
  	}
  	if i+1 == len(mirror.Endpoints) && endpointURLEqual(config.Default, &ep) {
  	// if the last endpoint is the default endpoint, move it there
  	config.Default = &ep
  	} else {
  	config.Endpoints = append(config.Endpoints, ep)
  	}
  	}
  	}
  	if host == "*" {
  	host = "_default"
  	}
  	hosts[host] = config
  	}
  	// Clean up hosts and default endpoints where resulting config leaves only defaults
  	for host, config := range hosts {
  	// if this host has no endpoints and the default has no config, delete this host
  	if len(config.Endpoints) == 0 && !endpointHasConfig(config.Default) {
  	delete(hosts, host)
  	}
  	}
  	return hosts
  	}

[brandond]

Duplicate of pinned issue: #9839

[kurokobo]

Ah sorry to bother you, thank you!