Create cluster failed when running talos with docker and mirror registry

[liangyuanpeng]

Hi teams, Talos is a great project and thanks for your work.

In my case, i run talos with docker and have set mirror registry for gcr.io and registry.k8s.io, and then the control plane is not working.

This node is can not access gcr.io and registry.k8s.io, and then i have test for the same command of talosctl cluster create --image ghcr.io/siderolabs/talos:v1.4.5 --install-image ghcr.io/siderolabs/installer:v1.4.5 --registry-mirror gcr.io=https://gcr.my.com --registry-mirror registry.k8s.io=https://registry.my.com. It's working, so i think it is about network and i have not idea to chech it.

talosctl logs:

...
bootstrapping cluster
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: OK
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: OK
◰ waiting for kubelet to be healthy: 1 error occurred:
◳ waiting for kubelet to be healthy: 1 error occurred:
    * 10.5.0.2: service "kubelet" not in expected state "Running": current state [Preparing] Running pre state

And the container is can not exec so i just got the error from container of talos-default-controlplane-1:

...
[talos] 2023/06/05 09:09:46 controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\n\terror getting node: Get \"https://localhost:6443/api/v1/nodes/talos-default-controlplane-1?timeout=30s\": dial tcp [::1]:6443: connect: connection refused"}
[talos] task startAllServices (1/1): 2023/06/05 09:10:01 service "kubelet" to be "up"
[talos] 2023/06/05 09:10:07 kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://10.5.0.2:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused"}
[talos] task startAllServices (1/1): 2023/06/05 09:10:16 service "kubelet" to be "up"
[talos] task startAllServices (1/1): 2023/06/05 09:10:31 service "kubelet" to be "up"
[talos] 2023/06/05 09:10:43 controller failed {"component": "controller-runtime", "controller": "k8s.ManifestApplyController", "error": "error creating mapping for object /v1/Secret/bootstrap-token-uqhto1: Get \"https://localhost:6443/api?timeout=32s\": dial tcp [::1]:6443: connect: connection refused"}
[talos] task startAllServices (1/1): 2023/06/05 09:10:46 service "kubelet" to be "up"
[talos] 2023/06/05 09:10:49 kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://10.5.0.2:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused"}
[talos] 2023/06/05 09:11:00 controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\n\terror getting node: Get \"https://localhost:6443/api/v1/nodes/talos-default-controlplane-1?timeout=30s\": dial tcp [::1]:6443: connect: connection refused"}

Note: mirror image registry is working, i can see the request for oci.

Remind me if i missed something,Thanks!

[liangyuanpeng]

Can see the kubelet is not working and get nothing from talos dmesg.

lan@lan:~/sshfiles$ talosctl -n 10.5.0.2 services
NODE       SERVICE      STATE       HEALTH   LAST CHANGE   LAST EVENT
10.5.0.2   apid         Running     OK       2m35s ago     Health check successful
10.5.0.2   containerd   Running     OK       2m36s ago     Health check successful
10.5.0.2   cri          Running     OK       2m35s ago     Health check successful
10.5.0.2   etcd         Running     OK       2m6s ago      Health check successful
10.5.0.2   kubelet      Preparing   ?        2m35s ago     Running pre state
10.5.0.2   machined     Running     OK       2m37s ago     Health check successful
10.5.0.2   trustd       Running     OK       2m35s ago     Health check successful
lan@lan:~/sshfiles$ talosctl -n 10.5.0.2 dmesg
...
10.5.0.2: kern:    info: [2023-06-05T09:51:58.724429943Z]: br-7ecd154289ea: port 2(veth492e430) entered blocking state
10.5.0.2: kern:    info: [2023-06-05T09:51:58.724431943Z]: br-7ecd154289ea: port 2(veth492e430) entered forwarding state
10.5.0.2: kern: warning: [2023-06-05T09:55:05.922642943Z]: mce: CPU2: Core temperature above threshold, cpu clock throttled (total events = 36593)
10.5.0.2: kern: warning: [2023-06-05T09:55:05.922643943Z]: mce: CPU0: Core temperature above threshold, cpu clock throttled (total events = 36593)
10.5.0.2: kern: warning: [2023-06-05T09:55:05.922644943Z]: mce: CPU1: Package temperature above threshold, cpu clock throttled (total events = 42950)
10.5.0.2: kern: warning: [2023-06-05T09:55:05.922646943Z]: mce: CPU3: Package temperature above threshold, cpu clock throttled (total events = 42950)
10.5.0.2: kern: warning: [2023-06-05T09:55:05.922647943Z]: mce: CPU0: Package temperature above threshold, cpu clock throttled (total events = 42950)
10.5.0.2: kern: warning: [2023-06-05T09:55:05.922649943Z]: mce: CPU2: Package temperature above threshold, cpu clock throttled (total events = 42950)
10.5.0.2: kern:    info: [2023-06-05T09:55:05.926649943Z]: mce: CPU0: Core temperature/speed normal
10.5.0.2: kern:    info: [2023-06-05T09:55:05.926650943Z]: mce: CPU2: Core temperature/speed normal
10.5.0.2: kern:    info: [2023-06-05T09:55:05.926651943Z]: mce: CPU3: Package temperature/speed normal
10.5.0.2: kern:    info: [2023-06-05T09:55:05.926653943Z]: mce: CPU1: Package temperature/speed normal
10.5.0.2: kern:    info: [2023-06-05T09:55:05.926653943Z]: mce: CPU2: Package temperature/speed normal
10.5.0.2: kern:    info: [2023-06-05T09:55:05.926655943Z]: mce: CPU0: Package temperature/speed normal

[smira]

The image pull error (if any) is in the talosctl dmesg output, try searching for retrying: line. It's not very well logged, but it's there.

[liangyuanpeng]

Thanks for your quickly reply and confirm, You're right. I follow the doc of https://www.talos.dev/v1.4/talos-guides/configuration/pull-through-cache/ and create cluster success.

[raydenz]

hello
I have the same problem
Can you give me more details on how to proceed to create your cluster with success
thank you very much
regards
Ray