Load Test for demo scenario for >10k Clusters

[ruanxin]

Implement a demo load test scenario, verify if the load test setup works as expected.

This task mainly have three parts:

1. Load test script

• Written as k6 script, a demo can be found here: https://github.com/ruanxin/performance-test/blob/main/load_generator.js
  • Load test the remote watching: watch resources on X remote clusters and ensure the reported customer changes are properly tracked and considered by the Kyma Operator.
  • TBC - done by PB already?: Implement test case to measure the amount of maximal watchable resources until K8s starts to deliver not all change events anymore => sync with @ebensom about limitations SREs wre already facing in their setup

2. Target Cluster under testing

• Provision 2 Gardener clusters, one as control plane, one as skr cluster
• Deploy prometheus-operator into control plane cluster
• Create a grafana dashboard, with some demo panel to monitor some predefined metrics
• Create a helm chart to define a dummy resource which simulate the behavior of resources defined in kyma components
  • https://github.com/ruanxin/performance-test/tree/main/kyma-load-test
• Deploy kyma-operator, manifest-operator into control plane cluster
• Deploy skr-operator into skr cluster which manage the reconciliation of target resources
• Config a remote OCI registery to save component descriptor files
• Run load test against control plane cluster, target 10000 kyma with 200000 modules
  • ⛔ 10% of the modules should be kcp modules NO kcp modules planned for Phase 1

3. Conclusion

• Present measuring / results to team => point out identified threshold

4. Findings and followup issue

• Refactoring watcher reconcile logic #266
• High memory consumption of module-manager module-manager#141
• ⛔ adjust configuartion based on load test finding module-manager#134
• fix: unsafe decoder for descriptor after webhook #283
• Refactor watcher and kyma reconcilers to improve load test performance #277
• fix: enable typed caching for sync-runner #284
• chore: configurable rate limit adjustments and increased perf test limits for >10k Clusters #282
• feat: refactor module catalog to sync into runtime #276
• feat: include pprof tracing into lifecycle-manager by flag #273
• feat: introduce (remote) client cache to reduce remote reconciliation latency #270
• feat: implement remote cluster info caching for remote clusters module-manager#145
• cache rendered manifest for helm module-manager#140
• adjust configuartion based on load test finding module-manager#134
• refactor: Adjustments for KCP watcher and load tests module-manager#132

Notes

• Rewrite manifest-operator to overwrite deployed helm chart resource name with kyma CR name as suffix
  • operator/internal/pkg/prepare/deploy.go:106
  • make sure the overwritten name is an unique identifier
• Load test should not run until listener, watcher integrated

[tobiscr]

Depends on kyma-project/runtime-watcher#70

[tobiscr]

Waiting for #200

[jakobmoellerdev]

Closing as only minor follow-ups need to be closed main round of load tests are considered complete