Fix aggregation of flaky test results in Jenkins #8472

npepinpe · 2021-12-23T09:39:21Z

Description

This PR fixes the aggregation of flaky test results. There was an issue where I changed the glob pattern which caused flaky tests to be ignored when collecting the reports. Essentially, the flaky test plugin adds a -FLAKY suffix to the test name and creates a new report for that run under surefire-reports or failsafe-reports, so we were missing a glob * at the end of the pattern.

Related issues

closes #8373

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

The changes are backwards compatibility with previous versions
If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/0.25) to the PR, in case that fails you need to create backports manually.

Testing:

There are unit/integration tests that verify all acceptance criterias of the issue
New tests are written to ensure backwards compatibility with further versions
The behavior is tested manually
The change has been verified by a QA run
The impact of the changes is verified by a benchmark

Documentation:

The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
New content is added to the release announcement

npepinpe · 2021-12-23T12:53:47Z

I've added 2 DO NOT MERGE commits which should be ignored, so please only review the first commit. These DO NOT MERGE commits are there to force flaky tests to occur and to run less tests. See the last build to see if/how it works 👍

Once reviewed, I'll rebase and remove the 2 commits.

korthout

LGTM 🎄

Just 1 question, but not blocking

Jenkinsfile

npepinpe · 2021-12-27T08:51:27Z

Good point - it might be safer to run it multiple times, and also clearer why/what we're doing.

Fixes the Jenkins junit plugin pattern used to detect test reports in our CI pipeline. The flaky test plugin will add a `-flaky` suffix, so we need to add an additional `*` to the pattern. Unfortunately as it's not a regex but simply an Ant glob pattern, there is no way to further restrict matches, so there's still the risk it could collide and double the test reports.

npepinpe · 2021-12-27T11:05:14Z

bors merge

8472: Fix aggregation of flaky test results in Jenkins r=npepinpe a=npepinpe ## Description This PR fixes the aggregation of flaky test results. There was an issue where I changed the glob pattern which caused flaky tests to be ignored when collecting the reports. Essentially, the flaky test plugin adds a `-FLAKY` suffix to the test name and creates a new report for that run under `surefire-reports` or `failsafe-reports`, so we were missing a glob `*` at the end of the pattern. ## Related issues closes #8373 Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com>

ghost · 2021-12-27T11:15:53Z

Build failed:

Run smoke tests on macos-latest

npepinpe · 2021-12-27T11:37:13Z

bors retry

ghost · 2021-12-27T12:30:44Z

Build succeeded:

github-actions · 2021-12-27T12:31:40Z

Backport failed for stable/1.1, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git fetch origin stable/1.1
git worktree add -d .worktree/backport-8472-to-stable/1.1 origin/stable/1.1
cd .worktree/backport-8472-to-stable/1.1
git checkout -b backport-8472-to-stable/1.1
ancref=$(git merge-base 5471fe471b009b4fc61d3f7b64fedf6fc731298a 76eb54eb993db5e67a78d4f3273537de060c8b88)
git cherry-pick -x $ancref..76eb54eb993db5e67a78d4f3273537de060c8b88

github-actions · 2021-12-27T12:31:42Z

Backport failed for stable/1.2, because it was unable to create a new branch.

Please cherry-pick the changes locally.

git fetch origin stable/1.2
git worktree add -d .worktree/backport-8472-to-stable/1.2 origin/stable/1.2
cd .worktree/backport-8472-to-stable/1.2
git checkout -b backport-8472-to-stable/1.2
ancref=$(git merge-base 5471fe471b009b4fc61d3f7b64fedf6fc731298a 76eb54eb993db5e67a78d4f3273537de060c8b88)
git cherry-pick -x $ancref..76eb54eb993db5e67a78d4f3273537de060c8b88

8483: [Snyk] Fix for 1 vulnerabilities r=npepinpe a=snyk-bot <h3>Snyk has created this PR to fix one or more vulnerable packages in the `maven` dependencies of this project.</h3> #### Changes included in this PR - Changes to the following files to upgrade the vulnerable dependencies to a fixed version: - parent/pom.xml #### Vulnerabilities that will be fixed ##### With an upgrade: Severity | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|:-------------------------|:-------------------------|:-------------------------|:------------------------- ![medium severity](https://res.cloudinary.com/snyk/image/upload/w_20,h_20/v1561977819/icon/m.png "medium severity") | Denial of Service (DoS) <br/>[SNYK-JAVA-COMFASTERXMLJACKSONCORE-2326698](https://snyk.io/vuln/SNYK-JAVA-COMFASTERXMLJACKSONCORE-2326698) | `org.springframework.boot:spring-boot-actuator-autoconfigure:` <br> `2.5.2 -> 2.5.8` <br> `org.springframework.boot:spring-boot-starter-web:` <br> `2.5.2 -> 2.5.8` <br> | No | No Known Exploit #### Vulnerabilities that could not be fixed - Upgrade: - Could not upgrade `io.camunda:zeebe-atomix-cluster@1.1.10-SNAPSHOT` to `io.camunda:zeebe-atomix-cluster@1.2.0`; Reason `could not apply upgrade, dependency is managed externally` ; Location: `provenance does not contain location` - Could not upgrade `io.camunda:zeebe-broker@1.1.10-SNAPSHOT` to `io.camunda:zeebe-broker@1.2.0`; Reason `could not apply upgrade, dependency is managed externally` ; Location: `provenance does not contain location` - Could not upgrade `io.camunda:zeebe-elasticsearch-exporter@1.1.10-SNAPSHOT` to `io.camunda:zeebe-elasticsearch-exporter@1.2.0`; Reason `could not apply upgrade, dependency is managed externally` ; Location: `provenance does not contain location` - Could not upgrade `io.camunda:zeebe-gateway@1.1.10-SNAPSHOT` to `io.camunda:zeebe-gateway@1.2.0`; Reason `could not apply upgrade, dependency is managed externally` ; Location: `provenance does not contain location` - Could not upgrade `io.camunda:zeebe-util@1.1.10-SNAPSHOT` to `io.camunda:zeebe-util@1.2.0`; Reason `could not apply upgrade, dependency is managed externally` ; Location: `provenance does not contain location` Check the changes in this PR to ensure they won't cause issues with your project. ------------ **Note:** *You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.* For more information: <img src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiI1ZDk1ZDkwMS1hOTZhLTQ4M2YtOGEzZS1jYTQ2OGI0ZmNkODUiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjVkOTVkOTAxLWE5NmEtNDgzZi04YTNlLWNhNDY4YjRmY2Q4NSJ9fQ==" width="0" height="0"/> 🧐 [View latest project report](https://app.snyk.io/org/zeebe-io/project/c2a97f6a-4bf7-4ff0-8ca8-b64956582654?utm_source=github&utm_medium=referral&page=fix-pr) 🛠 [Adjust project settings](https://app.snyk.io/org/zeebe-io/project/c2a97f6a-4bf7-4ff0-8ca8-b64956582654?utm_source=github&utm_medium=referral&page=fix-pr/settings) 📚 [Read more about Snyk's upgrade and patch logic](https://support.snyk.io/hc/en-us/articles/360003891078-Snyk-patches-to-fix-vulnerabilities) [//]: # (snyk:metadata:{"prId":"5d95d901-a96a-483f-8a3e-ca468b4fcd85","prPublicId":"5d95d901-a96a-483f-8a3e-ca468b4fcd85","dependencies":[{"name":"io.camunda:zeebe-atomix-cluster","from":"1.1.10-SNAPSHOT","to":"1.2.0"},{"name":"io.camunda:zeebe-broker","from":"1.1.10-SNAPSHOT","to":"1.2.0"},{"name":"io.camunda:zeebe-elasticsearch-exporter","from":"1.1.10-SNAPSHOT","to":"1.2.0"},{"name":"io.camunda:zeebe-gateway","from":"1.1.10-SNAPSHOT","to":"1.2.0"},{"name":"io.camunda:zeebe-util","from":"1.1.10-SNAPSHOT","to":"1.2.0"},{"name":"org.springframework.boot:spring-boot-actuator-autoconfigure","from":"2.5.2","to":"2.5.8"},{"name":"org.springframework.boot:spring-boot-starter-web","from":"2.5.2","to":"2.5.8"}],"packageManager":"maven","projectPublicId":"c2a97f6a-4bf7-4ff0-8ca8-b64956582654","projectUrl":"https://app.snyk.io/org/zeebe-io/project/c2a97f6a-4bf7-4ff0-8ca8-b64956582654?utm_source=github&utm_medium=referral&page=fix-pr","type":"auto","patch":[],"vulns":["SNYK-JAVA-COMFASTERXMLJACKSONCORE-2326698"],"upgrade":["SNYK-JAVA-COMFASTERXMLJACKSONCORE-2326698"],"isBreakingChange":false,"env":"prod","prType":"fix","templateVariants":[],"priorityScoreList":[null]}) 8487: [Backport stable/1.1] Fix junit report pattern r=deepthidevaki a=npepinpe ## Description Fixes the Jenkins junit plugin pattern used to detect test reports in our CI pipeline. The flaky test plugin will add a `-flaky` suffix, so we need to add an additional `*` to the pattern. Unfortunately as it's not a regex but simply an Ant glob pattern, there is no way to further restrict matches, so there's still the risk it could collide and double the test reports. (cherry picked from commit 76eb54e) ## Related issues relates to #8472 Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com>

8488: [Backport stable/1.2] Fix junit report pattern r=deepthidevaki a=npepinpe ## Description Fixes the Jenkins junit plugin pattern used to detect test reports in our CI pipeline. The flaky test plugin will add a `-flaky` suffix, so we need to add an additional `*` to the pattern. Unfortunately as it's not a regex but simply an Ant glob pattern, there is no way to further restrict matches, so there's still the risk it could collide and double the test reports. (cherry picked from commit 76eb54e) ## Related issues relates to #8472 Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com>

npepinpe requested a review from korthout December 23, 2021 12:51

npepinpe changed the title ~~8373 fix junit pattern~~ Fix aggregation of flaky test results in Jenkins Dec 23, 2021

npepinpe marked this pull request as ready for review December 23, 2021 12:53

npepinpe added backport stable/1.1 labels Dec 23, 2021

korthout approved these changes Dec 23, 2021

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

korthout mentioned this pull request Dec 24, 2021

[Backport stable/1.2] Add recreate benchmark script #8477

Merged

npepinpe force-pushed the 8373-fix-junit-pattern branch 3 times, most recently from 4e8d4c7 to 76eb54e Compare December 27, 2021 11:05

ghost merged commit 22b408d into develop Dec 27, 2021

ghost deleted the 8373-fix-junit-pattern branch December 27, 2021 12:30

This was referenced Dec 28, 2021

[Backport stable/1.1] Fix junit report pattern #8487

Merged

[Backport stable/1.2] Fix junit report pattern #8488

Merged

npepinpe added Release: 1.1.10 labels Dec 29, 2021

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix aggregation of flaky test results in Jenkins #8472

Fix aggregation of flaky test results in Jenkins #8472

npepinpe commented Dec 23, 2021 •

edited

Loading

npepinpe commented Dec 23, 2021 •

edited

Loading

korthout left a comment

npepinpe commented Dec 27, 2021

npepinpe commented Dec 27, 2021

ghost commented Dec 27, 2021

npepinpe commented Dec 27, 2021

ghost commented Dec 27, 2021

github-actions bot commented Dec 27, 2021

github-actions bot commented Dec 27, 2021

Fix aggregation of flaky test results in Jenkins #8472

Fix aggregation of flaky test results in Jenkins #8472

Conversation

npepinpe commented Dec 23, 2021 • edited Loading

Description

Related issues

Definition of Done

npepinpe commented Dec 23, 2021 • edited Loading

korthout left a comment

Choose a reason for hiding this comment

npepinpe commented Dec 27, 2021

npepinpe commented Dec 27, 2021

ghost commented Dec 27, 2021

npepinpe commented Dec 27, 2021

ghost commented Dec 27, 2021

github-actions bot commented Dec 27, 2021

github-actions bot commented Dec 27, 2021

npepinpe commented Dec 23, 2021 •

edited

Loading

npepinpe commented Dec 23, 2021 •

edited

Loading