Automation of Rosa Scaling Benchmark (#444): Part 2

[andyuk1986]

PR contains:

1. Created 2 new actions for performing prometheus queries and calculating/storing result numbers into JSON file;
2. besides the result json file I am publishing the intermediate files with metric numbers before /after benchmark execution, for being able to debug the correctness of numbers before merging the PR;
3. I have added "continue-on-error": true to benchmark execution steps, as sometimes Gatling gives failure during report generation (if some of conditions doesn't met) and the whole workflow fails, even though the metrics can be taken and can be calculated
4. fixed if conditions on action steps as the steps were working no matter if condition was met or not;

The workflow execution for 5 minutes gives the following JSON:

{
  "Testing memory for creating sessions": {
    "500 MB Memory per pod in 3 node cluster handles:": "114831 Active User Sessions"
  },
  "Testing CPU usage for user logins": {
    "1vCPU per pod in 3 node cluster handles:": "45 User Logins Per Second"
  },
  "Testing CPU usage for client credential grants": {
    "1vCPU per pod in 3 node cluster handles:": "478 Client Credential Grants Per Second"
  }
}

Closes #444

[ahus1]

@andyuk1986 - thank you for preparing calculation of the results.

I pushed some small code syntax change, I hope they don't break anything while making the code a bit simpler to read. The # language=bash prefix instructs IntelliJ to do syntax highlighting in the following key.

The calculations look good for the requests per seconds for both users and clients looks good.

The calculation for memory is probably off: The CRITERIA_VALUE used here is the entities created in the realm. The correct value to use IMHO would be the number of sessions created. An approximation would be a "users per second logging in" multiplied by "measurement time in seconds". This is also available as a metrics from Prometheus, so one could read it before and after the run, but that would be a bit more work. Please choose either way, and update the calculations.

Looking at the JSON, this is a JSON I found in one of your runs:

{
  "Testing memory for creating sessions": {
    "500 MB Memory per pod in 3 node cluster handles:": "171997 Active User Sessions"
  },
  "Testing CPU usage for user logins": {
    "1vCPU per pod in 3 node cluster handles:": "47 User Logins Per Second"
  },
  "Testing CPU usage for client credential grants": {
    "1vCPU per pod in 3 node cluster handles:": "486 Client Credential Grants Per Second"
  }
}

I ask you to make the result a simple value so it can be used in reports. It shouldn't be quoted as it is a number, not a string.

Due to that, the naming changed a little, and I also tried to shorten it.

{
  "Memory usage for sessions": {
    "Active sessions per 500 MB memory per Pod in 3 Pod cluster": 171997
  },
  "CPU usage for user logins": {
    "User Logins per second per 1vCPU per Pod in a 3 Pod cluster": 47
  },
  "CPU usage for client credential grants": {
    "Client Credential Grants per second per 1vCPU per Pod in a 3 Pod cluster": 486
  }
}

Please let me know if you have questions.

[andyuk1986]

@ahus1 hi, thanks a lot for the review. I have updated PR with suggested changes.
Currently the new JSON file looks like (example for benchmarks run in 5 minutes):

{
  "Memory usage for sessions": {
    "Active sessions per 500 MB memory per Pod in 3 Pod cluster": "143116"
  },
  "CPU usage for user logins": {
    "User Logins per second per 1vCPU per Pod in 3 Pod cluster": "37"
  },
  "CPU usage for client credential grants": {
    "Client Credential Grants per second per 1vCPU per Pod in 3 Pod cluster": "427"
  }
}

[ahus1]

Thank you for this contribution. Its great that we now have this automation to calculate the metrics by the click of a button within one hour.

I did two small changes:

• Adding a |tonumber to the jq command to have the value printed without quotes, so it can be processed in future steps without converting it from a string to a number
• Masking the value of OC_TOKEN so it doesn't show up in the console.