Load test failing on dedicated runner

[codeboten]

Component(s)

No response

Describe the issue you're reporting

The following failure occurs when running the load test on dedicated runners:

2023/10/04 17:53:04 CPU consumption is 78.3%, max expected is 60%
    test_case.go:292: CPU consumption is 78.3%, max expected is 60%
    scenarios.go:304: 
        	Error Trace:	/home/ghrunner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/tests/scenarios.go:304
        	            				/home/ghrunner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/tests/trace_test.go:191
        	Error:      	"50" is not less than "0"
        	Test:       	TestTraceNoBackend10kSPS/MemoryLimit
2023/10/04 17:53:04 Stopped generator. Sent:     29090 items
--- FAIL: TestTraceNoBackend10kSPS (18.48s)
    --- PASS: TestTraceNoBackend10kSPS/NoMemoryLimit (15.28s)
    --- FAIL: TestTraceNoBackend10kSPS/MemoryLimit (3.20s)

https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/6408839554/job/17401287975

[codeboten]

The numbers reported by the dedicated runner pretty different from what we're seeing on github shared runners. A sample below shows the results on shared runners:

2023/10/04 13:46:35 Writing Agent log to /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/tests/results/TestMetric10kDPS/OpenCensus/agent.log
2023/10/04 13:46:35 Agent running, pid=2091
2023/10/04 13:46:35 Starting load generator at 10000 items/sec.
2023/10/04 13:46:38 Agent RAM (RES):   0 MiB, CPU: 0.0% | Sent:    204400 items | Received:   203,700 items (67,897/sec)
2023/10/04 13:46:41 Agent RAM (RES):  79 MiB, CPU:39.0% | Sent:    414400 items | Received:   413,700 items (68,947/sec)
2023/10/04 13:46:44 Agent RAM (RES):  79 MiB, CPU:39.7% | Sent:    624057 items | Received:   623,700 items (69,301/sec)
2023/10/04 13:46:47 Agent RAM (RES):  78 MiB, CPU:39.7% | Sent:    834400 items | Received:   833,700 items (69,477/sec)
2023/10/04 13:46:50 Agent RAM (RES):  79 MiB, CPU:38.7% | Sent:   1044400 items | Received: 1,043,700 items (69,578/sec)
2023/10/04 13:46:50 Stopped generator. Sent:   1049300 items

And on the dedicated runner:

 2023/10/04 21:40:23 Writing Agent log to /home/ghrunner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/tests/results/TestMetric10kDPS/OpenCensus/agent.log
2023/10/04 21:40:23 Agent running, pid=1283643
2023/10/04 21:40:23 Starting load generator at 10000 items/sec.
2023/10/04 21:40:26 Agent RAM (RES):  32 MiB, CPU: 0.0% | Sent:    204400 items | Received:   203,700 items (67,887/sec)
2023/10/04 21:40:29 Agent RAM (RES):  76 MiB, CPU:64.3% | Sent:    413700 items | Received:   413,700 items (68,951/sec)
2023/10/04 21:40:32 Agent RAM (RES):  76 MiB, CPU:68.0% | Sent:    623779 items | Received:   623,700 items (69,302/sec)
2023/10/04 21:40:35 Agent RAM (RES):  76 MiB, CPU:69.3% | Sent:    833700 items | Received:   833,000 items (69,418/sec)
2023/10/04 21:40:38 Agent RAM (RES):  77 MiB, CPU:70.0% | Sent:   1044400 items | Received: 1,043,000 items (69,533/sec)

The dedicated hardware runs with 16 cores, compared to the 2 cores for shared runners. The question I have is should the testbed be modified for set the number of CPUs used by the tests? Or should the expected values of the tests match the new hardware?

[codeboten]

One way to set number of CPU in the childProcessCollector's Start func:

cp.cmd.Env = append(os.Environ(),
		"GOMAXPROCS=2",
	)

[codeboten]

Fwiw, the change in performance reporting will impact the reported benchmarks:

[djaglowski]

Setting the number of cores seems like a good idea. Can a runner run multiple jobs simultaneously? Assuming so, cores may end up being restricted inconsistently based on the number of jobs running.

[codeboten]

Can a runner run multiple jobs simultaneously?

As far as I can tell, the runners currently only run one job at a time now, but you're right that it could change in the future.