Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test detection is broken #8373

Closed
npepinpe opened this issue Dec 13, 2021 · 2 comments · Fixed by #8472
Closed

Flaky test detection is broken #8373

npepinpe opened this issue Dec 13, 2021 · 2 comments · Fixed by #8472
Assignees
@npepinpe
Labels
kind/bug Categorizes an issue or PR as a bug

Comments

@npepinpe
Copy link
Member

Describe the bug

Some builds report flaky tests, but no failing test cases are reported in Jenkins. It's unclear whether or not the issue is with the plugin or with our Jenkins pipeline.

To Reproduce

Not sure. Here's an example build: https://ci.zeebe.camunda.cloud/blue/organizations/jenkins/camunda-cloud%2Fzeebe/detail/develop/1171/pipeline. I've saved the build forever, please delete it once this is fixed.

Expected behavior

Flaky tests are properly reported as separate failures for easier debugging.
@npepinpe npepinpe added kind/bug Categorizes an issue or PR as a bug Impact: Testing labels Dec 13, 2021
@npepinpe npepinpe added this to Ready in Zeebe Dec 15, 2021
@npepinpe npepinpe self-assigned this Dec 15, 2021
@npepinpe
Copy link
Member Author

As this has an impact on our day to day, let's fix it soon - I also think I have an idea what the issue is.

@npepinpe
Copy link
Member Author

I think the issue is that we changed (and by we, I mean, myself) the pattern when grepping for results. The flaky test plugin adds a -flaky suffix to the tests, but the pattern is now "**/*/TEST*${SUREFIRE_REPORT_NAME_SUFFIX}.xml", so it won't match it. Unfortunately this isn't a regex, but just the Ant glob pattern, so it's not very powerful. The reason for the change was to avoid collisions and having patterns too permissive, which sometimes resulted in test results being picked up by the wrong stage.

@npepinpe npepinpe moved this from Ready to Planned in Zeebe Dec 23, 2021
@npepinpe npepinpe moved this from Planned to In progress in Zeebe Dec 23, 2021
@npepinpe npepinpe mentioned this issue Dec 23, 2021
Merged
9 tasks
@npepinpe npepinpe moved this from In progress to Review in progress in Zeebe Dec 23, 2021
ghost pushed a commit that referenced this issue Dec 27, 2021
@npepinpe
merge: #8472
a0ae64c 
8472: Fix aggregation of flaky test results in Jenkins r=npepinpe a=npepinpe

## Description

This PR fixes the aggregation of flaky test results. There was an issue where I changed the glob pattern which caused flaky tests to be ignored when collecting the reports. Essentially, the flaky test plugin adds a `-FLAKY` suffix to the test name and creates a new report for that run under `surefire-reports` or `failsafe-reports`, so we were missing a glob `*` at the end of the pattern.

## Related issues

closes #8373 



Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com>
@ghost ghost closed this as completed in 22b408d Dec 27, 2021
Zeebe automation moved this from Review in progress to Done Dec 27, 2021
@KerstinHebel KerstinHebel removed this from Done in Zeebe Mar 23, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@npepinpe npepinpe

Labels
kind/bug Categorizes an issue or PR as a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix aggregation of flaky test results in Jenkins camunda/camunda
3 participants
@npepinpe @remcowesterhoud @pihme