Scale and Performance testing for 1.26 k8s clusters

[vivek-shilimkar]

We need to do performance testing regarding large clusters to see how it performs..

Specific usecases:

• Local and downstream 1.26 RKE1 and RKE2 clusters - individual cluster performance.
• Large number of downstream RKE1/RKE2 Rancher provisioned clusters.

Individual Cluster Performance

Rancher Cluster Config:

Rancher version: latest 2.7.5-rc1
Rancher cluster type: RKE1/RKE2 HA (3 all-roles nodes, 1 worker node tainted to run rancher-monitoring)
Cluster Size: Medium
Docker version: 20.10
K8s version: v1.26.4-rancher2-1/v1.26.4+rke2r1
Log level: debug
certmanager version: 1.11.0

Downstream Cluster Config:

Downstream cluster types: HA RKE1, RKE2, K3s (3 all-roles nodes)
Installed apps: N/A
Deployments: ranchertest/mytestcontainer
rke1 version = v1.26.4-rancher2-1
rke2 version = v1.26.4+rke2r1
k3s version = v1.26.4+k3s1

Method:

1. Setup HA Local (k8s version) cluster with Rancher
   1. Install monitoring
   2. Install other workloads/Apps if desired (SKIPPED)
2. Provision HA Downstream (k8s version) clusters (1 of each RKE1, RKE2, K3s)
3. Load each downstream cluster with a set of "objects"
   1. 100 secrets
   2. 10 projects
   3. 12 namespaces
   4. 300 users (users are created in the Local cluster)
   5. 1 project Role Template
   6. 300 project Role Template bindings
4. Deploy desired pods onto each downstream cluster
   1. Deploy ranchertest/mytestcontainer until reaching 330 pod limit
   2. Collect CPU and Memory profile logs as scale-in progresses
   3. Collect rancher pod logs as scale-in progresses
5. Collect rancher-monitoring metrics artifacts

Scalability

Rancher Cluster Config:

Rancher version: latest 2.7.5-rc1
Rancher cluster type: RKE1/RKE2 HA , 3 all-roles nodes, 1 worker node (tainted to run rancher-monitoring)
Cluster Size: Medium
Docker version: 20.10
K8s version: v1.26.4-rancher2-1/v1.26.4+rke2r1
Log level: debug
certmanager version: 1.11.0

Downstream Cluster Config:

Downstream cluster types: RKE1/RKE2 (1 all-roles nodes)
Installed apps: N/A
Deployments: N/A
rke1 version = v1.26.4-rancher2-1
rke2 version = v1.26.4+rke2r1

Method:

1. Setup HA Local (k8s version) cluster with Rancher
   1. Install monitoring
   2. Install other workloads/Apps if desired (SKIPPED)
2. Let idle for 30 minutes
3. Provision Downstream (k8s version) clusters (RKE1/RKE2)
   • Scale to half the documented cluster limit
4. Collect rancher-monitoring metrics artifacts and review for unexpected behavior
5. Provision Downstream (k8s version) clusters (RKE1/RKE2)
   • Scale to the full documented cluster limit
6. Collect rancher-monitoring metrics artifacts and review for unexpected behavior

[git-ival]

Was blocked due to lack of PSACT support via the rancher2 tfp, see rancher/terraform-provider-rancher2#1112 and rancher/terraform-provider-rancher2#908.

Unblocked by creating a PR for V1 clusters then combining the following PRs and building the provider locally:

• Enable setting default PSACT for rancher2_cluster add admission_configuration support with state migration logic terraform-provider-rancher2#1119
• feat: add PSACT to cluster_v2 resource and data-source terraform-provider-rancher2#1117

Testing is underway, RKE1 is completed

[git-ival]

Results have been logged internally