failed to register machine inventory: read elemental configuration: websocket: close 1006 (abnormal closure): unexpected EOF

[kkaempf]

Elemental registration fails as indicated in $title.

Environment

• Rancher 2.6.8 running under Docker
• openSUSE Tumbleweed
• Elemental ISO booted under virt-manager

Full log from elemental-register --debug /run/initramfs/live

time="2022-09-19T13:11:27Z" level=info msg="Register version 0.5.0, commit 846c610, commit date git20220912"
time="2022-09-19T13:11:27Z" level=debug msg="scanning config path /run/initramfs/live"
time="2022-09-19T13:11:27Z" level=info msg="reading config file /run/initramfs/live/livecd-cloud-config.yaml"
time="2022-09-19T13:11:27Z" level=debug msg="input config:\nconfig.Config{\n  Elemental: config.Elemental{\n    Install: config.Install{\n      Firmware: \"\",\n      Device: \"\",\n      NoFormat: false,\n      ConfigURLs: nil,\n      ISO: \"\",\n      SystemURI: \"\",\n      Debug: false,\n      TTY: \"\",\n      PowerOff: false,\n      Reboot: false,\n      EjectCD: false,\n    },\n    Registration: config.Registration{\n      URL: \"https://172.17.0.2/elemental/registration/9jw6l9pnnzgxwfxwmbtzfdgz7c5l2q7bg6f6q7x2ssptbw8v4vqn8l\",\n      CACert: \"-----BEGIN CERTIFICATE-----\\nMIIBvjCCAWOgAwIBAgIBADAKBggqhkjOPQQDAjBGMRwwGgYDVQQKExNkeW5hbWlj\\nbGlzdGVuZXItb3JnMSYwJAYDVQQDDB1keW5hbWljbGlzdGVuZXItY2FAMTY2MzA3\\nMTcyMjAeFw0yMjA5MTMxMjIyMDJaFw0zMjA5MTAxMjIyMDJaMEYxHDAaBgNVBAoT\\nE2R5bmFtaWNsaXN0ZW5lci1vcmcxJjAkBgNVBAMMHWR5bmFtaWNsaXN0ZW5lci1j\\nYUAxNjYzMDcxNzIyMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE6LQIKsVQH1OA\\ntEWsaKv7pIvLb9vvpeTWeOPvtuMV/IK+ehLX/V7imhdWEzRlYxMgNa0JmXWFDPJM\\nJzIrpwVBm6NCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYD\\nVR0OBBYEFDfEFcrw2S1ufehq9f8EkTQzWVfIMAoGCCqGSM49BAMCA0kAMEYCIQCK\\ndeeZzeNjYX96BVlLXELozE6w3lii00OlGo9vgGYnAgIhANuExC0JgByzquJdGpA/\\nZViaNOxKfeu9Qd1CYUQdQVt2\\n-----END CERTIFICATE-----\",\n      EmulateTPM: false,\n      EmulatedTPMSeed: 1,\n      NoSMBIOS: false,\n    },\n    SystemAgent: config.SystemAgent{\n      URL: \"\",\n      Token: \"\",\n      SecretName: \"\",\n      SecretNamespace: \"\",\n    },\n  },\n  CloudConfig: map[string]interface {}(nil),\n}"
time="2022-09-19T13:11:27Z" level=info msg="Using TPMHash 1c9b6e73145c651a5cb73ee0eed13c2300effb030f4b7dbfefe5e7a63db96308 to dial wss://172.17.0.2/elemental/registration/9jw6l9pnnzgxwfxwmbtzfdgz7c5l2q7bg6f6q7x2ssptbw8v4vqn8l"
time="2022-09-19T13:11:27Z" level=debug msg="start TPM attestation"
time="2022-09-19T13:11:27Z" level=info msg="TPM attestation successful"
time="2022-09-19T13:11:27Z" level=debug msg="send SMBIOS data"
time="2022-09-19T13:11:27Z" level=debug msg="get elemental configuration"
time="2022-09-19T13:11:27Z" level=error msg="failed to register machine inventory: read elemental configuration: websocket: close 1006 (abnormal closure): unexpected EOF"
time="2022-09-19T13:11:32Z" level=info msg="Using TPMHash 1c9b6e73145c651a5cb73ee0eed13c2300effb030f4b7dbfefe5e7a63db96308 to dial wss://172.17.0.2/elemental/registration/9jw6l9pnnzgxwfxwmbtzfdgz7c5l2q7bg6f6q7x2ssptbw8v4vqn8l"
time="2022-09-19T13:11:32Z" level=debug msg="start TPM attestation"
time="2022-09-19T13:11:32Z" level=info msg="TPM attestation successful"
time="2022-09-19T13:11:32Z" level=debug msg="send SMBIOS data"
time="2022-09-19T13:11:32Z" level=debug msg="get elemental configuration"
time="2022-09-19T13:11:32Z" level=error msg="failed to register machine inventory: read elemental configuration: websocket: close 1006 (abnormal closure): unexpected EOF"

Full log from elemental-operator (copied from k9s)

time="2022-09-19T12:58:15Z" level=info msg="Operator version 0.5.0, commit 846c610, commit date git20220912"                                                                               │
│ time="2022-09-19T12:58:15Z" level=info msg="Starting controller at namespace cattle-elemental-system. Upgrade sync interval at: 1h0m0s"                                                    │
│ time="2022-09-19T12:58:15Z" level=info msg="Applying CRD managedosimages.elemental.cattle.io"                                                                                              │
│ time="2022-09-19T12:58:15Z" level=info msg="Applying CRD machineinventories.elemental.cattle.io"                                                                                           │
│ time="2022-09-19T12:58:15Z" level=info msg="Applying CRD machineregistrations.elemental.cattle.io"                                                                                         │
│ time="2022-09-19T12:58:15Z" level=info msg="Applying CRD managedosversions.elemental.cattle.io"                                                                                            │
│ time="2022-09-19T12:58:15Z" level=info msg="Applying CRD managedosversionchannels.elemental.cattle.io"                                                                                     │
│ time="2022-09-19T12:58:15Z" level=info msg="Applying CRD machineinventoryselectors.elemental.cattle.io"                                                                                    │
│ time="2022-09-19T12:58:15Z" level=info msg="Applying CRD machineinventoryselectortemplates.elemental.cattle.io"                                                                            │
│ time="2022-09-19T12:58:16Z" level=info msg="Creating event broadcaster for mos-bundle"                                                                                                     │
│ time="2022-09-19T12:58:16Z" level=info msg="Creating event broadcaster for machine-registration"                                                                                           │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting elemental.cattle.io/v1beta1, Kind=MachineRegistration controller"                                                                     │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting cluster.x-k8s.io/v1beta1, Kind=Machine controller"                                                                                    │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting /v1, Kind=Secret controller"                                                                                                          │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting fleet.cattle.io/v1alpha1, Kind=Bundle controller"                                                                                     │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting steve aggregation client"                                                                                                             │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting elemental.cattle.io/v1beta1, Kind=ManagedOSImage controller"                                                                          │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting elemental.cattle.io/v1beta1, Kind=ManagedOSVersion controller"                                                                        │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting elemental.cattle.io/v1beta1, Kind=MachineInventorySelector controller"                                                                │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting elemental.cattle.io/v1beta1, Kind=MachineInventory controller"                                                                        │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting elemental.cattle.io/v1beta1, Kind=ManagedOSVersionChannel controller"                                                                 │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting /v1, Kind=ServiceAccount controller"                                                                                                  │
│ time="2022-09-19T12:58:16Z" level=info msg="Starting management.cattle.io/v3, Kind=Setting controller"                                                                                     ```

[fgiudici]

Update: tested on a KVM single node kubernetes v1.23 K3s cluster, installed Rancher Manager v2.6.8, Operator version 0.5.0 commit 846c610. Client registration worked fine.
Tried the same with Rancher Manager v2.6.8 on docker: reproduced the issue --> the container there spawns a kubernetes v1.24 k3s single node cluster.
So, seems the issue is only present when using Rancher Manager under docker.
The issue is in the operator when using kubernetes 1.24.
Upgrading K3s on the KVM setup reproduced the issue there too.

The root cause of the failure is that the service account used by the operator has no secrets associated.
This is likely due to the way we generate the ServiceAccount in the operator, no more supported in Kubernetes 1.24.
As the error was not properly managed no meaningful log was displayed on the operator side, which closed the communication without actually sending the elemental config. I have a simple patch managing the error, will push with the actual fix I'm currently working onto, which will involve review of the Service Account management/generation.

[agracey]

When you have a fix available, I can also test it as I'm running into the same issue

[kkaempf]

Confirmed fixed, thank you !