Investigate automation of ROSA scaling benchmark

[ahus1]

Understand and discuss how this can be automated

https://www.keycloak.org/keycloak-benchmark/benchmark-guide/latest/report/rosa-benchmark-key-results

[andyuk1986]

So after running/understanding the benchmark and it's steps, having chat with Alexander, gathering here the info discussed:

1. The first step would be to put all the steps into a GitHub action which could run against the shared cluster gh-keycloak. The tricky part is (which is not yet automated) would be to read the metrics from OpenShift once the test is complete. The document could then be update manually first, and later automatically.
2. Start with a single data point (like 200 users per second for authorization code). We would later add more data points (like 50 and 300) to show that the resource utilization is linear.
3. The script "dataset-import.sh" uses a special URL to create data on the server. There is a second URL which allows a minimal interaction with Keycloak internal caches, for example clearing them, checking if an entry exists, or showing the number of entries. This might come in handy in the later steps of this test. And we'll use it also for our Active/Passive tests to verify if an entry exists on the other datacenter.

[andyuk1986]

PR for first part of implementation is sent: #462

[andyuk1986]

PR for second and final part of implementation is sent: #492

So what has been done:

1. Workflow is created for running 3 keycloak benchmarks on created EC2 instances for measuring different parameters (vCPU, memory) for user logins per second, credential grants per second and user sessions.
2. The actions are created which are retrieving corresponding Prometheus metrics before and after the benchmarks are running and then calculate the numbers of supported criterias (user logins per second, credential grants per second and user sessions) for current environment.
3. The JSON file is generated and archived in the end of the workflow, which stores all calculated data. An example of this json document is (example json calculated for benchmarks running for 5 minutes):

{
  "Testing memory for creating sessions": {
    "500 MB Memory per pod in 3 node cluster handles:": "114831 Active User Sessions"
  },
  "Testing CPU usage for user logins": {
    "1vCPU per pod in 3 node cluster handles:": "45 User Logins Per Second"
  },
  "Testing CPU usage for client credential grants": {
    "1vCPU per pod in 3 node cluster handles:": "478 Client Credential Grants Per Second"
  }
}

4. The workflow also archives the Gatling aggregat report and stats.json files.