-
Notifications
You must be signed in to change notification settings - Fork 9.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
testgrid-analysis: create issues based on test set flakiness instead of individual tests. #17924
base: main
Are you sure you want to change the base?
Conversation
Skipping CI for Draft Pull Request. |
This report looks useful. However, I'm not sure if having it as a GitHub issue makes sense. I feel like we'll just need to close the report issues, which may add to the work of managing issues. That being said, I agree that there's great value in having the reports more visible rather than nested in a GitHub action run. My suggestion would be to explore Job Summaries. Also, something that would be useful if we decide to use Job Summaries, would be to provide the link to the GitHub action run in the generated issue. cc. @serathius, as you suggested more visibility of the reports. |
/retest |
How spammy do we expect this new issue opening to be? We already have quite a few open flake issues https://github.com/etcd-io/etcd/issues?q=is%3Aopen+is%3Aissue+label%3Atype%2Fflake We might end up having to continually close issues as duplicates until some existing known flakes are resolved which could be annoying. Some description of what impact this pr is expected to have would help make a decision on if it will be helpful or harmful. |
Signed-off-by: Siyuan Zhang <sizhang@google.com>
I will re-evaluate if this PR is necessary. Mark it as draft for now. |
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
@siyuanfoundation: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.
#17773 (comment)
Tested locally, example issue: siyuanfoundation#64
If a test set is flaky, e.g. pull-request unit test fails > 10% of the time, it would create a parent issue for the test set, and create sub-issues for the top 5 fail tests.
If the test keep failing, it would append new comment every time the script is run (which is daily), without creating new issues.