Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testgrid-analysis: create issues based on test set flakiness instead of individual tests. #17924

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

siyuanfoundation
Copy link
Contributor

@siyuanfoundation siyuanfoundation commented May 1, 2024

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

#17773 (comment)

Tested locally, example issue: siyuanfoundation#64

If a test set is flaky, e.g. pull-request unit test fails > 10% of the time, it would create a parent issue for the test set, and create sub-issues for the top 5 fail tests.
If the test keep failing, it would append new comment every time the script is run (which is daily), without creating new issues.

@k8s-ci-robot
Copy link

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ahrtr
Copy link
Member

ahrtr commented May 3, 2024

cc @ArkaSaha30 @jmhbnz @ivanvc

@ivanvc
Copy link
Member

ivanvc commented May 3, 2024

This report looks useful. However, I'm not sure if having it as a GitHub issue makes sense. I feel like we'll just need to close the report issues, which may add to the work of managing issues.

That being said, I agree that there's great value in having the reports more visible rather than nested in a GitHub action run. My suggestion would be to explore Job Summaries. Also, something that would be useful if we decide to use Job Summaries, would be to provide the link to the GitHub action run in the generated issue.

cc. @serathius, as you suggested more visibility of the reports.

@jmhbnz
Copy link
Member

jmhbnz commented May 3, 2024

/retest

@jmhbnz
Copy link
Member

jmhbnz commented May 3, 2024

How spammy do we expect this new issue opening to be? We already have quite a few open flake issues https://github.com/etcd-io/etcd/issues?q=is%3Aopen+is%3Aissue+label%3Atype%2Fflake

We might end up having to continually close issues as duplicates until some existing known flakes are resolved which could be annoying.

Some description of what impact this pr is expected to have would help make a decision on if it will be helpful or harmful.

Signed-off-by: Siyuan Zhang <sizhang@google.com>
@siyuanfoundation
Copy link
Contributor Author

I will re-evaluate if this PR is necessary. Mark it as draft for now.

@k8s-ci-robot
Copy link

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot
Copy link

@siyuanfoundation: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-etcd-unit-test-amd64 edc5f16 link true /test pull-etcd-unit-test-amd64
pull-etcd-unit-test-arm64 edc5f16 link true /test pull-etcd-unit-test-arm64

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

6 participants