Unable to provision K3S downstream cluster in Rancher

[mdrahman-suse]

Environmental Info:
K3s Version:

v1.27.13-rc1+k3s1

Node(s) CPU architecture, OS, and Version:

Ubuntu 22.04 or Sles 15 sp4

Cluster Configuration:

3 servers (Deployed Rancher), All roles or Split roles downstream cluster

Describe the bug:

Mentioned here: #10001 (comment)

Steps To Reproduce:

• Installed K3s as single or HA setup
• Deploy latest Rancher v2.7.12 or v2.8.3
• Once the cluster is up, access Rancher UI and provision downstream cluster with
  • All roles
  • Split roles

Expected behavior:

• Downstream cluster is provisioned successfully

Actual behavior:

• Downstream cluster provisioning fails
  • Rancher UI shows the servers in Waiting to connect... state
  • From capi-controller-manager-* log

2024/04/22 19:52:09 [INFO] [planner] rkecluster fleet-default/mdk3s128sp: configuring bootstrap node(s) mdk3s128sp-pool1-6659f4969x4snws-kg5zq: waiting for agent to check in and apply initial plan
2024/04/22 19:52:09 [INFO] [planner] rkecluster fleet-default/mdk3s128sp: mdk3s128sp-pool1-6659f4969x4snws-kg5zq
2024/04/22 19:52:10 [INFO] [planner] rkecluster fleet-default/mdk3s128sp: non-ready bootstrap machine(s) mdk3s128sp-pool1-6659f4969x4snws-kg5zq and join url to be available on bootstrap node

Additional context / logs:

• Error observed on Rancher deployed cluster node

Apr 22 16:17:22 k3s[2750]: E0422 16:17:22.798129    2750 cacher.go:469] cacher (machinedeployments.cluster.x-k8s.io): unexpected ListAndWatch error: failed to list cluster.x-k8s.io/v1alpha4, Kind=MachineDeployment: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=MachineDeployment failed: Post "https://capi-webhook-service.cattle-provisioning-capi-system.svc:443/convert?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority; reinitializing...

[mdrahman-suse]

Validation on master branch with commit 94e29e2

Environment and config

Sles 15 sp4
1 etcd, 1 cp, 1 worker

Testing

• Install k3s on a split role cluster
• Ensure the cluster is up
• Perform the curl from etcd node
  • to localhost, should return data with nodeIP
  • to cp-only node, should return data with 403 Forbidden

Replication

~> k3s -v
k3s version v1.29.3+k3s1 (8aecc26b)
go version go1.21.8

• localhost

~> curl -ks https://localhost:6443/db/info
{"members":[{"ID":17663870494773375509,"name":"ip-xxx-xx-xx-140.us-east-2.compute.internal-2cb028bb","peerURLs":["https://xxx.xx.xx.140:2380"],"clientURLs":["https://xxx.xx.xx.140:2379"]}]}

• cp-only node

~> curl -ks https://<cp-hostname or ip>:6443/db/info
{"members":[{"ID":17663870494773375509,"name":"ip-xxx-xx-xx-140.us-east-2.compute.internal-2cb028bb","peerURLs":["https://xxx.xx.xx.140:2380"],"clientURLs":["https://xxx.xx.xx.140:2379"]}]}

Validation

~> k3s -v
k3s version v1.29.4+k3s-94e29e2e (94e29e2e)
go version go1.21.9

• localhost

~> curl -ks https://localhost:6443/db/info
{"members":[{"ID":16136583629868037345,"name":"ip-xxx-xx-x-55.us-east-2.compute.internal-cdf0642f","peerURLs":["https://xxx.xx.x.55:2380"],"clientURLs":["https://xxx.xx.x.55:2379"]}]}

• cp-only node

~> curl -ks https://<cp-hostname or ip>:6443/db/info
{
  "kind": "Status",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden",
  "reason": "Forbidden",
  "code": 403
}

Note: Not able to validate in Rancher as this version is not supported yet. Will validate on supported versions with Rancher