Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Preserve output of flaky tests #10109

Closed
npepinpe opened this issue Aug 17, 2022 · 0 comments · Fixed by #10520
Closed

Preserve output of flaky tests #10109

npepinpe opened this issue Aug 17, 2022 · 0 comments · Fixed by #10520
Assignees
@npepinpe
Labels
kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. version:8.2.0-alpha1 Marks an issue as being completely or in parts released in 8.2.0-alpha1 version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0

Comments

@npepinpe
Copy link
Member

Description

When we re-run tests due to flakiness, it seems we lose the output file associated. You can see in failed runs that while there's two XML files for the test (one suffixed with -FLAKY), we only have the successful run's output. This makes diagnosing flaky tests difficult unless we're lucky and they are easy to reproduce.

To save us time, we should figure out how to preserve the output of the test between re-runs. Maybe a test listener can do this, or anything which somehow gets notified that a test will run again.
@npepinpe npepinpe added the kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. label Aug 17, 2022
@npepinpe npepinpe self-assigned this Aug 18, 2022
@npepinpe npepinpe mentioned this issue Sep 27, 2022
Merged
10 tasks
@korthout korthout added the version:8.2.0-alpha1 Marks an issue as being completely or in parts released in 8.2.0-alpha1 label Nov 1, 2022
@npepinpe npepinpe added the version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0 label Apr 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@npepinpe npepinpe

Labels
kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. version:8.2.0-alpha1 Marks an issue as being completely or in parts released in 8.2.0-alpha1 version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Preserve output of flaky test runs camunda/zeebe
2 participants
@npepinpe @korthout