Perfkit runs are always "official" while running them with Jenkins #18805
Description
What is "official" flag?
"Official" flag is a PerfkitBenchmarker boolean flag that is set in Jenkins. Perfkit uses it to determine whether test results should be treated as official ones. Currently, in the whole Performance Testing Framework it means that such results will be:
- displayed in the Performance Testing Dashboards (https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688)
- used for detecting anomalies in those dashboards and reporting them to the community (this is work in progress at the time of writing)
How can Performance tests be run right now?
Currently, we have two options for running performance tests:
- run them periodically (this is done by Jenkins on master branch 4 times a day)
- trigger them on demand from Pull Request - for example by typing "Run Java TextIO Performance Test" on Github comment in the PR. For now, they also use master for building the code, which is useless in terms of running them in the PR. They should use the branches code instead otherwise it's misguiding for the user and error-prone. This issue is addressed here: https://issues.apache.org/jira/browse/BEAM-4140
What is the problem?
We shouldn't mark on demand runs as "official", because those are run to test unmerged code. This will pollute the dashboards with unwanted results on the plots. in the future, when anomaly detection is merged, it will discover false-positive anomalies (based on unmerged code).
Proposed solution:
The official flag in Jenkins should be set based on "GIT_BRANCH" environment variable (AFAIK it's out of the box in Jenkins). Only results from master should be official.
Imported from Jira BEAM-4298. Original Jira may contain additional context.
Reported by: ŁukaszG.