Run benchmarks of memory usage and provide recommended value of memory limit

[randmonkey]

Is there an existing issue for this?

• I have searched the existing issues

Does this enhancement require public documentation?

• I have added an Acceptance Criteria item for adding and/or adjusting public documentation (if applicable)

Problem Statement

In FTI-5854, customer has asked about the increase of memory usage from KIC 2.8 to KIC 3.1, and the 256Mi (which is in comments of values.yaml of charts) limit is not sufficient for them.
We found that we do not have document about memory usage of KIC and recommended value of memory limits.

Proposed Solution

• Run benchmarks to get the memory required when KIC controls different amounts of k8s resources
  • Scenarios can be similar to them in performance tests
• Acknowledge the major components that consume significant amount of memory
  • Maybe optimize memory usage by disabling controllers not used
• Add document to provide recommendations of allocating memory to KIC containers in different scenarios

Acceptance Criteria

• Have a benchmark method and result about memory usage, reflecting the relationship between memory usage and controlled k8s resources
• Have the document to provide recommended values of memory limits for KIC containers in different scenarios

[czeslavo]

I agree that having a benchmark reporting memory consumption is something we should build. I imagine that having charts tracking the memory consumption for a few basic scenarios (10, 1k, 1kk of resources managed) would be enough for us as input to update the suggested memory requirements in docs before we release another version of KIC.

I'd extract a separate issue for "Acknowledge the major components that consume significant amount of memory" though as it can be an independent task - we can get along without it at the beginning at least. If we see that after some changes the reported memory consumption grows, we can profile locally to get better insights.

I think that one more tricky aspect of memory consumption is the potential of goroutine leaks which are hard to detect without having a long-running KIC instance (having a team-managed instance of KGO with a Gateway deploying KIC nightly and monitoring its memory/CPU consumption would be already a good step forward in that matter). We could also make use of a tool like https://github.com/uber-go/goleak in our tests to make sure that after integration tests teardown, the manager doesn't leave any leaked goroutines.