Enforcing correct CI workers for benchmarks

[eyalkoren]

Context: benchmarks script failing on new CI processor - https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-java%2Fapm-agent-java-mbp/detail/main/285/pipeline:

CPU_MODEL='12th Gen Intel(R) Core(TM) i5-12500 '
...
Cannot determine base frequency for CPU model [12th Gen Intel(R) Core(TM) i5-12500 ]. Please adjust the build script.

Based on Intel's processor's spec.

@kuisathaverat Do you think a ci:benchmarks label to tell CI to run benchmarks for PRs would be useful?

[eyalkoren]

Based on the other types, it seems to count the effective threads rather than physical cores, but I am not sure

[eyalkoren]

run benchmark tests

[ghost]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-07-05T18:39:44.993+0000

• Duration: 50 min 6 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3052
Skipped	36
Total	3088

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run benchmark tests : Run the benchmark tests.

• run jdk compatibility tests : Run the JDK Compatibility tests.

• run integration tests : Run the Agent Integration tests.

• run end-to-end tests : Run the APM-ITs.

• run windows tests : Build & tests on windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[felixbarny]

Why are the benchmarks suddenly executed on a new hardware? This would make any comparisons with past results meaningless.

[eyalkoren]

Why are the benchmarks suddenly executed on a new hardware? This would make any comparisons with past results meaningless.

Does this mean we cannot upgrade? Or is it enough to manually compare now and start accumulating new history?

[felixbarny]

I suppose it doesn't mean that we can never upgrade but we should carefully review when doing so. For example, is the new worker a dedicated bare metal machine like the old one or is it a cloud VM that may have noisy neighbours? Which background processes are running on the new worker that might interfere with the benchmark?

[felixbarny]

@cachedout do you know why the benchmarks aren't executed on the dedicated bare metal machines anymore?

[kuisathaverat]

Do you think a ci:benchmarks label to tell CI to run benchmarks for PRs would be useful?

I am not a fan of using labels to trigger CI jobs, I preferred it to be intentional using a GitHub comment because these additional jobs usually do not need to run on every commit, and if you do it you will waisting CI time.

cc @v1v

[v1v]

Let me look at what's going on

[v1v]

Do you think a ci:benchmarks label to tell CI to run benchmarks for PRs would be useful?

It's not supported,

apm-agent-java/Jenkinsfile

Lines 57 to 58 in 32ee207

	// disabled by default, not required for merge
	booleanParam(name: 'bench_ci', defaultValue: false, description: 'Enable benchmarks')

You can enable it by using something like:

                expression { matchesPrLabel(label: 'ci:benchmarks') }

in

apm-agent-java/Jenkinsfile

Lines 300 to 304 in 32ee207

	anyOf {
	branch 'main'
	expression { return env.GITHUB_COMMENT?.contains('benchmark tests') }
	expression { return params.bench_ci }
	}

do you know why the benchmarks aren't executed on the dedicated bare metal machines anymore?

It uses metal and, recently there are a bunch of new CI workers for being used by the prodfiler.

Unfortunately, the metal label was added in the CI provisioner, and the CI Jenkinsifle for this project should use something else to be sure it uses the dedicated ones.

What baremetals can you use?

• linux&&metal
• benchmarks <--- that matches one worker -> worker-1213919 therefore I wonder if this is the one to be used instead and it's consumed by the load-testing as per https://github.com/elastic/apm-agent-java/pull/1467/files#diff-7b74176d56dcf2d85162c7e15a233c3115eca7999fc6744a9b3e59bda5a2861cR65

[v1v]

@eyalkoren , I just added some changes to this PR so:

• I reverted the change for supporting a new CPU
• Fixed the agent labels to filter only those workers that were used in the past.
• Supported ci:benchmarks tag
• Delete the workspace in the baremetals

Let's see how it goes, I didn't want to change the description of this PR, feel free to do it so 🙇

[cachedout]

It uses metal and, recently there are a bunch of new CI workers for being used by the prodfiler.

raises hand This is my fault. I requested workers for the prodfiler benchmarking project and had them labeled with a new label but missed the fact that they'd also be put in an existing pool. Apologies.