Problem Running Talos Control Plane In Kubernetes Pod

[infinitydon]

Hi,

I am currently trying to run talos control plane as PODs instead of a VM/Baremetal/Docker, I am using a statefulset shown below:

apiVersion: v1
kind: Service
metadata:
  name: talos-eben-svc
  namespace: talos-eben
  labels:
    app: talos-eben-cp
spec:
  type: LoadBalancer
  selector:
    app: talos-eben-cp
  ports:
  - name: k8s-port
    protocol: TCP
    port: 6443
    targetPort: k8s-port
  - name: talos-api
    protocol: TCP
    port: 50000
    targetPort: talos-api
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: talos-eben-cp
  namespace: talos-eben
spec:
  serviceName: talos-eben-svc
  replicas: 3
  selector:
    matchLabels:
      app: talos-eben-cp
  template:
    metadata:
      labels:
        app: talos-eben-cp        
    spec:
      containers:
      - name: talos-cp
        image: ghcr.io/siderolabs/talos:v1.6.3
        securityContext:
          privileged: true        
        env:
        - name: PLATFORM
          value: "container"        
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 50000
          name: talos-api
        - containerPort: 6443
          name: k8s-port
        volumeMounts:
        - mountPath: /system/state
          name: system-state
        - mountPath: /var
          name: var
        - mountPath: /etc/cni
          name: etc-cni
        - mountPath: /etc/kubernetes
          name: etc-kubernetes
        - mountPath: /usr/libexec/libexec
          name: libexec
        - mountPath: /usr/etc/udev
          name: etc-udev 
        - mountPath: /opt
          name: opt
        - mountPath: /run
          name: run
        - mountPath: /system
          name: system
        - mountPath: /tmp
          name: tmp                                                                                    
      volumes:
      - name: run
        emptyDir: {}
      - name: system
        emptyDir: {}
      - name: tmp
        emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      name: system-state
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-path"
      resources:
        requests:
          storage: 3Gi
  - metadata:
      name: var
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-path"
      resources:
        requests:
          storage: 3Gi
  - metadata:
      name: etc-cni
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-path"
      resources:
        requests:
          storage: 3Gi
  - metadata:
      name: etc-kubernetes
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-path"
      resources:
        requests:
          storage: 3Gi
  - metadata:
      name: libexec
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-path"
      resources:
        requests:
          storage: 3Gi
  - metadata:
      name: etc-udev
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-path"
      resources:
        requests:
          storage: 3Gi
  - metadata:
      name: opt
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-path"
      resources:
        requests:
          storage: 3Gi

I am using the following commands to install the cluster (CONTROL_PLANE_IP -- This is the LoadBalancer IP from the kubernetes service):

kubectl -n talos-eben get svc
NAME             TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                          AGE
talos-eben-svc   LoadBalancer   10.43.190.226   192.168.100.152   6443:30079/TCP,50000:32262/TCP   18m

kubectl -n talos-eben get po -o wide
NAME              READY   STATUS             RESTARTS        AGE   IP            NODE               NOMINATED NODE   READINESS GATES
talos-eben-cp-0   0/1     CrashLoopBackOff   6 (5m ago)      20m   10.42.0.230   o5gs-nephio-mgmt   <none>           <none>
talos-eben-cp-1   0/1     CrashLoopBackOff   6 (3m36s ago)   19m   10.42.0.238   o5gs-nephio-mgmt   <none>           <none>
talos-eben-cp-2   0/1     CrashLoopBackOff   6 (3m14s ago)   19m   10.42.0.246   o5gs-nephio-mgmt   <none>           <none>


export CONTROL_PLANE_IP=192.168.100.152
talosctl gen config talos-vbox-cluster https://$CONTROL_PLANE_IP:6443 --output-dir _out

talosctl apply-config --insecure --nodes 10.42.0.230 --file _out/controlplane.yaml
talosctl apply-config --insecure --nodes 10.42.0.238 --file _out/controlplane.yaml
talosctl apply-config --insecure --nodes 10.42.0.246 --file _out/controlplane.yaml

As shown above all the pods are conitually crashing, logs from one of the pods:

kubectl -n talos-eben logs talos-eben-cp-0 -f
2024/02/06 17:46:18 limited GOMAXPROCS to 4
[talos] 2024/02/06 17:46:18 initialize sequence: 4 phase(s)
[talos] 2024/02/06 17:46:18 phase systemRequirements (1/4): 1 tasks(s)
[talos] 2024/02/06 17:46:18 task setupSystemDirectory (1/1): starting
[talos] 2024/02/06 17:46:18 task setupSystemDirectory (1/1): done, 82.971µs
[talos] 2024/02/06 17:46:18 phase systemRequirements (1/4): done, 222.086µs
[talos] 2024/02/06 17:46:18 phase etc (2/4): 3 tasks(s)
[talos] 2024/02/06 17:46:18 task setUserEnvVars (3/3): starting
[talos] 2024/02/06 17:46:18 task createOSReleaseFile (2/3): starting
[talos] 2024/02/06 17:46:18 task setUserEnvVars (3/3): done, 41.129µs
[talos] 2024/02/06 17:46:18 task CreateSystemCgroups (1/3): starting
[talos] 2024/02/06 17:46:18 task createOSReleaseFile (2/3): done, 627.738µs
[talos] 2024/02/06 17:46:18 nftables chains updated {"component": "controller-runtime", "controller": "network.NfTablesChainController", "chains": []}
[talos] 2024/02/06 17:46:18 node identity established {"component": "controller-runtime", "controller": "cluster.NodeIdentityController", "node_id": "Gmjpretc4ggIPdnGUQTYHuj1vZLiuBHNK1tk6rKU9KF"}
[talos] 2024/02/06 17:46:18 setting resolvers {"component": "controller-runtime", "controller": "network.ResolverSpecController", "resolvers": ["1.1.1.1", "8.8.8.8"]}
[talos] 2024/02/06 17:46:18 setting resolvers {"component": "controller-runtime", "controller": "network.ResolverSpecController", "resolvers": ["1.1.1.1", "8.8.8.8"]}
[talos] 2024/02/06 17:46:18 setting time servers {"component": "controller-runtime", "controller": "network.TimeServerSpecController", "addresses": ["pool.ntp.org"]}
[talos] 2024/02/06 17:46:18 setting time servers {"component": "controller-runtime", "controller": "network.TimeServerSpecController", "addresses": ["pool.ntp.org"]}
[talos] 2024/02/06 17:46:18 ethtool watcher failed to start {"component": "controller-runtime", "controller": "network.LinkStatusController", "error": "error getting family information for ethtool: netlink receive: no such file or directory"}
[talos] 2024/02/06 17:46:18 error dialing ethtool socket {"component": "controller-runtime", "controller": "network.LinkStatusController", "error": "netlink receive: no such file or directory"}
[talos] 2024/02/06 17:46:18 task CreateSystemCgroups (1/3): done, 39.743963ms
[talos] 2024/02/06 17:46:18 phase etc (2/4): done, 39.816687ms
[talos] 2024/02/06 17:46:18 phase machined (3/4): 2 tasks(s)
[talos] 2024/02/06 17:46:18 task startContainerd (2/2): starting
[talos] 2024/02/06 17:46:18 task startMachined (1/2): starting
[talos] 2024/02/06 17:46:18 service[containerd](Preparing): Running pre state
[talos] 2024/02/06 17:46:18 service[containerd](Preparing): Creating service runner
[talos] 2024/02/06 17:46:18 TPM device is not available, skipping PCR extension
[talos] 2024/02/06 17:46:18 service[machined](Preparing): Running pre state
[talos] 2024/02/06 17:46:18 service[machined](Preparing): Creating service runner
[talos] 2024/02/06 17:46:18 service[machined](Running): Service started as goroutine
[talos] 2024/02/06 17:46:18 service[containerd](Running): Process Process(["/bin/containerd" "--address" "/system/run/containerd/containerd.sock" "--state" "/system/run/containerd" "--root" "/system/var/lib/containerd"]) started with PID 17
[talos] 2024/02/06 17:46:19 service[machined](Running): Health check successful
[talos] 2024/02/06 17:46:19 task startMachined (1/2): done, 1.001743329s
[talos] 2024/02/06 17:46:19 service[containerd](Running): Health check successful
[talos] 2024/02/06 17:46:19 task startContainerd (2/2): done, 1.004220656s
[talos] 2024/02/06 17:46:19 phase machined (3/4): done, 1.004243296s
[talos] 2024/02/06 17:46:19 phase config (4/4): 1 tasks(s)
[talos] 2024/02/06 17:46:19 task loadConfig (1/1): starting
[talos] 2024/02/06 17:46:19 machine config loaded successfully {"component": "controller-runtime", "controller": "config.AcquireController", "sources": ["state"]}
[talos] 2024/02/06 17:46:19 task loadConfig (1/1): done, 5.507141ms
[talos] 2024/02/06 17:46:19 phase config (4/4): done, 5.52698ms
[talos] 2024/02/06 17:46:19 initialize sequence: done: 1.049861587s
[talos] 2024/02/06 17:46:19 install sequence: 0 phase(s)
[talos] 2024/02/06 17:46:19 install sequence: done: 3.004µs
[talos] 2024/02/06 17:46:19 boot sequence: 11 phase(s)
[talos] 2024/02/06 17:46:19 phase saveConfig (1/11): 1 tasks(s)
[talos] 2024/02/06 17:46:19 task saveConfig (1/1): starting
[talos] 2024/02/06 17:46:19 task saveConfig (1/1): done, 407.103µs
[talos] 2024/02/06 17:46:19 phase saveConfig (1/11): done, 454.258µs
[talos] 2024/02/06 17:46:19 phase memorySizeCheck (2/11): 1 tasks(s)
[talos] 2024/02/06 17:46:19 task memorySizeCheck (1/1): starting
[talos] 2024/02/06 17:46:19 skipping memory size check in the container
[talos] 2024/02/06 17:46:19 task memorySizeCheck (1/1): done, 100.787µs
[talos] 2024/02/06 17:46:19 phase memorySizeCheck (2/11): done, 133.689µs
[talos] 2024/02/06 17:46:19 phase diskSizeCheck (3/11): 1 tasks(s)
[talos] 2024/02/06 17:46:19 task diskSizeCheck (1/1): starting
[talos] 2024/02/06 17:46:19 skipping disk size check in the container
[talos] 2024/02/06 17:46:19 task diskSizeCheck (1/1): done, 8.356µs
[talos] 2024/02/06 17:46:19 phase diskSizeCheck (3/11): done, 25.347µs
[talos] 2024/02/06 17:46:19 phase env (4/11): 1 tasks(s)
[talos] 2024/02/06 17:46:19 task setUserEnvVars (1/1): starting
[talos] 2024/02/06 17:46:19 task setUserEnvVars (1/1): done, 6.948µs
[talos] 2024/02/06 17:46:19 phase env (4/11): done, 27.999µs
[talos] 2024/02/06 17:46:19 phase dbus (5/11): 1 tasks(s)
[talos] 2024/02/06 17:46:19 task startDBus (1/1): starting
[talos] 2024/02/06 17:46:19 service[apid](Waiting): Waiting for service "containerd" to be "up", api certificates
[talos] 2024/02/06 17:46:19 task startDBus (1/1): done, 745.678µs
[talos] 2024/02/06 17:46:19 phase dbus (5/11): done, 789.488µs
[talos] 2024/02/06 17:46:19 phase sharedFilesystems (6/11): 1 tasks(s)
[talos] 2024/02/06 17:46:19 task setupSharedFilesystems (1/1): starting
[talos] 2024/02/06 17:46:19 task setupSharedFilesystems (1/1): done, 36.063µs
[talos] 2024/02/06 17:46:19 phase sharedFilesystems (6/11): done, 49.729µs
[talos] 2024/02/06 17:46:19 phase var (7/11): 1 tasks(s)
[talos] 2024/02/06 17:46:19 task setupVarDirectory (1/1): starting
[talos] 2024/02/06 17:46:19 task setupVarDirectory (1/1): done, 63.782µs
[talos] 2024/02/06 17:46:19 phase var (7/11): done, 84.072µs
[talos] 2024/02/06 17:46:19 phase legacyCleanup (8/11): 1 tasks(s)
[talos] 2024/02/06 17:46:19 task cleanupLegacyStaticPodFiles (1/1): starting
[talos] 2024/02/06 17:46:19 task cleanupLegacyStaticPodFiles (1/1): failed: error opening manifests directory: open /etc/kubernetes/manifests: no such file or directory
[talos] 2024/02/06 17:46:19 phase legacyCleanup (8/11): failed
[talos] 2024/02/06 17:46:19 boot sequence: failed
[talos] 2024/02/06 17:46:19 service[apid](Failed): Condition failed: context canceled
[talos] 2024/02/06 17:46:19 service[machined](Finished): Service finished successfully
[talos] 2024/02/06 17:46:19 service[containerd](Stopping): Sending SIGTERM to Process(["/bin/containerd" "--address" "/system/run/containerd/containerd.sock" "--state" "/system/run/containerd" "--root" "/system/var/lib/containerd"])
[talos] 2024/02/06 17:46:19 kubeprism KubePrism is enabled {"component": "controller-runtime", "controller": "k8s.KubePrismController", "endpoint": "127.0.0.1:7445"}
[talos] 2024/02/06 17:46:19 service[containerd](Finished): Service finished successfully
[talos] 2024/02/06 17:46:19 error running phase 8 in boot sequence: task 1/1: failed, error opening manifests directory: open /etc/kubernetes/manifests: no such file or directory
[talos] 2024/02/06 17:46:19 failed destroying service resource {"component": "controller-runtime", "controller": "v1alpha1.ServiceController", "id": "containerd", "error": "destroy finalizer rate limited: context canceled"}
[talos] 2024/02/06 17:46:19 kubeprism KubePrism is disabled {"component": "controller-runtime", "controller": "k8s.KubePrismController"}
[talos] 2024/02/06 17:46:20 controller runtime finished
[talos] 2024/02/06 17:46:29 rebooting in 10 seconds
[talos] 2024/02/06 17:46:30 rebooting in 9 seconds
[talos] 2024/02/06 17:46:31 rebooting in 8 seconds
[talos] 2024/02/06 17:46:32 rebooting in 7 seconds
[talos] 2024/02/06 17:46:33 rebooting in 6 seconds
[talos] 2024/02/06 17:46:34 rebooting in 5 seconds
[talos] 2024/02/06 17:46:35 rebooting in 4 seconds
[talos] 2024/02/06 17:46:36 rebooting in 3 seconds
[talos] 2024/02/06 17:46:37 rebooting in 2 seconds
[talos] 2024/02/06 17:46:38 rebooting in 1 seconds
[talos] 2024/02/06 17:46:39 rebooting in 0 seconds
[talos] 2024/02/06 17:46:40 unmounted /opt (/dev/sda1)
[talos] 2024/02/06 17:46:40 failed unmounting /var: device or resource busy
[talos] 2024/02/06 17:46:40 unmounted /tmp (/dev/sda1)
[talos] 2024/02/06 17:46:40 failed unmounting /system: device or resource busy
[talos] 2024/02/06 17:46:40 failed unmounting /run: device or resource busy
[talos] 2024/02/06 17:46:40 unmounted /dev/termination-log (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/kubernetes (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/cni (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/hosts (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /system/state (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/hostname (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/resolv.conf (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /usr/etc/udev (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /usr/libexec/kubernetes (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/os-release (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/cri/conf.d/hosts (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/cri/conf.d/01-registries.part (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/machine-id (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/cri/conf.d/cri.toml (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/hosts (/dev/sda1)
[talos] 2024/02/06 17:46:40 unmounted /etc/resolv.conf (/dev/sda1)
[talos] 2024/02/06 17:46:40 retrying 3 unmount operations...
[talos] 2024/02/06 17:46:41 failed unmounting /var: device or resource busy
[talos] 2024/02/06 17:46:41 failed unmounting /system: device or resource busy
[talos] 2024/02/06 17:46:41 failed unmounting /run: device or resource busy
[talos] 2024/02/06 17:46:41 retrying 3 unmount operations...
[talos] 2024/02/06 17:46:42 failed unmounting /var: device or resource busy
[talos] 2024/02/06 17:46:42 failed unmounting /system: device or resource busy
[talos] 2024/02/06 17:46:42 failed unmounting /run: device or resource busy
[talos] 2024/02/06 17:46:42 retrying 3 unmount operations...
[talos] 2024/02/06 17:46:43 failed unmounting /var: device or resource busy
[talos] 2024/02/06 17:46:43 failed unmounting /system: device or resource busy
[talos] 2024/02/06 17:46:43 failed unmounting /run: device or resource busy
[talos] 2024/02/06 17:46:43 retrying 3 unmount operations...
[talos] 2024/02/06 17:46:44 failed unmounting /var: device or resource busy
[talos] 2024/02/06 17:46:44 failed unmounting /system: device or resource busy
[talos] 2024/02/06 17:46:44 failed unmounting /run: device or resource busy
[talos] 2024/02/06 17:46:44 retrying 3 unmount operations...
[talos] 2024/02/06 17:46:45 failed unmounting /var: device or resource busy
[talos] 2024/02/06 17:46:45 failed unmounting /system: device or resource busy
[talos] 2024/02/06 17:46:45 failed unmounting /run: device or resource busy
[talos] 2024/02/06 17:46:45 retrying 3 unmount operations...
[talos] 2024/02/06 17:46:46 failed unmounting /var: device or resource busy
[talos] 2024/02/06 17:46:46 failed unmounting /system: device or resource busy
[talos] 2024/02/06 17:46:46 failed unmounting /run: device or resource busy
[talos] 2024/02/06 17:46:46 retrying 3 unmount operations...
[talos] 2024/02/06 17:46:47 failed unmounting /var: device or resource busy
[talos] 2024/02/06 17:46:47 failed unmounting /system: device or resource busy
[talos] 2024/02/06 17:46:47 failed unmounting /run: device or resource busy
[talos] 2024/02/06 17:46:47 retrying 3 unmount operations...
[talos] 2024/02/06 17:46:48 failed unmounting /var: device or resource busy
[talos] 2024/02/06 17:46:48 failed unmounting /system: device or resource busy
[talos] 2024/02/06 17:46:48 failed unmounting /run: device or resource busy
[talos] 2024/02/06 17:46:48 retrying 3 unmount operations...
[talos] 2024/02/06 17:46:49 failed unmounting /var: device or resource busy
[talos] 2024/02/06 17:46:49 failed unmounting /system: device or resource busy
[talos] 2024/02/06 17:46:49 failed unmounting /run: device or resource busy
[talos] 2024/02/06 17:46:49 retrying 3 unmount operations...
[talos] 2024/02/06 17:46:50 waiting for sync...
[talos] 2024/02/06 17:46:51 sync done

Currently running talosctl v1.6.3:

talosctl version
Client:
        Tag:         v1.6.3
        SHA:         d53e07c1
        Built:
        Go version:  go1.21.6 X:loopvar
        OS/Arch:     linux/amd64

Any idea what could be wrong?

Thanks

[smira]

It is most probably a bug in Talos, but you can try to workaround it by mounting even emptyDir mount to /etc/kubernetes/manifest

[infinitydon]

Just tried that but there was no difference, the PODs are still crashing with the same error logs

[infinitydon]

Fixed the control-plane pods by adding an init-container to create the /etc/kubernetes/manifests folder:

      initContainers:
      - name: volume-dirs-init-cntr
        image: busybox:1.35
        command:
        - "/bin/mkdir"
        args:
        - "-p"
        - "/etc/kubernetes/manifests"
        volumeMounts:
        - mountPath: /etc/kubernetes
          name: etc-kubernetes

kubectl -n talos-eben get po -o wide
NAME              READY   STATUS    RESTARTS   AGE   IP            NODE               NOMINATED NODE   READINESS GATES
talos-eben-cp-0   1/1     Running   0          15m   10.42.0.111   o5gs-nephio-mgmt   <none>           <none>
talos-eben-cp-1   1/1     Running   0          14m   10.42.0.119   o5gs-nephio-mgmt   <none>           <none>
talos-eben-cp-2   1/1     Running   0          13m   10.42.0.128   o5gs-nephio-mgmt   <none>           <none>


kubectl --kubeconfig /tmp/temp-talos-k8s-cp/admin.conf get no -o wide
NAME              STATUS   ROLES           AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION      CONTAINER-RUNTIME
talos-eben-cp-0   Ready    control-plane   4m3s    v1.29.1   10.42.0.111       <none>        Talos (v1.6.3)   5.4.0-170-generic   containerd://1.7.11
talos-eben-cp-1   Ready    control-plane   2m28s   v1.29.1   10.42.0.119       <none>        Talos (v1.6.3)   5.4.0-170-generic   containerd://1.7.11
talos-eben-cp-2   Ready    control-plane   111s    v1.29.1   10.42.0.128       <none>        Talos (v1.6.3)   5.4.0-170-generic   containerd://1.7.11
talos-ph5-n7t     Ready    <none>          2m55s   v1.29.1   192.168.100.212   <none>        Talos (v1.6.3)   6.1.74-talos        containerd://1.7.11

But creating test deployment is failing, complains about flannel missing directory:

 kubectl --kubeconfig /tmp/temp-talos-k8s-cp/admin.conf describe po test-7955cf7657-p7qgj
Name:             test-7955cf7657-p7qgj
Namespace:        default
Priority:         0
Service Account:  default
Node:             talos-ph5-n7t/192.168.100.212
Start Time:       Tue, 06 Feb 2024 22:09:25 +0000
Labels:           app=test
                  pod-template-hash=7955cf7657
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Controlled By:    ReplicaSet/test-7955cf7657
Containers:
  nginx:
    Container ID:
    Image:          nginx
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6wfzw (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  kube-api-access-6wfzw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               12s   default-scheduler  Successfully assigned default/test-7955cf7657-p7qgj to talos-ph5-n7t
  Warning  FailedCreatePodSandBox  11s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0067c2c42970d6cfdf717827e3d7d1c5d58d4d215e3378eac62b0f88c0b1dbad": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory

Worker node is on a proxmox VM, do I have to somehow create this folder in the worker node?

[infinitydon]

Fixed the flannel error, it was due to the IP address config between the PODs and VM network. All looks good now

@smira - Will the /etc/kubernetes/manifest directory error be fixed in the talos code? I think talosctl cluster create --wait (Docker cluster creation) must be creating the directory also before initializing the Talos control-plane. Let me know your thought.

Thanks

[smira]

yes, it's a bug either way, need to see how to do a proper fix and test here.

[smira]

a fix will be in Talos 1.7+, and backported to Talos 1.6.5