Baseline Testing: Longevity

[mpstefan]

Ensure NKG does not leak resources after X number of days under normal conditions.

Acceptance

• A test process is documented to test for longevity by building an environment with NKG and an application. The application is called through NKG consistently and resources are added or removed over the course of 3 days.
• Scripts are created to simulate traffic and resource changes over 3 days.
• Errors reported by NKG, failures in traffic, and saturation metrics (memory, CPU, disk) are reported as the result of this test.
• No element of the test is randomized and the environment is consistent.
• The test is carried out manually and the results are reported in a static document within the NKG repo.

[vepatel]

• We should decide on a fixed configuration before running these tests(eg: CPU, Memory, Nodes, Number of replicas), and keep it same while running these for future versions,
• imo "resource changes over 3 days" fits scale testing better