Skip to content

fix a flakey test #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
GrantBirki opened this issue Apr 15, 2025 · 0 comments · Fixed by #23
Closed

fix a flakey test #22

GrantBirki opened this issue Apr 15, 2025 · 0 comments · Fixed by #23
Assignees
Labels
bug Something isn't working

Comments

@GrantBirki
Copy link
Member

GrantBirki commented Apr 15, 2025

Occasionally, it looks like the following test fails:

$ script/test
╭────────┬─────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────────────────────────╮
│ Status │ Elapsed │                                                                                  Test                                                                                  │          Package           │
├────────┼─────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────────────────────────┤
│  PASS  │  0.01   │ TestParseRepositories                                                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestCreatePullRequest                                                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestValidateLabels                                                                                                                                                     │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch                                                                                                                                                        │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestPrMatchesCriteriaWithMocks                                                                                                                                         │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestDisplayTableStats                                                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestDisplayJSONStats                                                                                                                                                   │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestDisplayPlainStats                                                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestApplyDefaultOwner                                                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestApplyDefaultOwner/empty_default_owner                                                                                                                              │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestApplyDefaultOwner/repo_already_has_owner                                                                                                                           │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestApplyDefaultOwner/repo_needs_default_owner                                                                                                                         │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositories/empty_inputs                                                                                                                                     │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositories/single_repository_arg                                                                                                                            │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositories/multiple_repository_args                                                                                                                         │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositories/comma-separated_repository_args                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositories/with_default_owner                                                                                                                               │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositories/with_valid_file                                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositories/with_invalid_file                                                                                                                                │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositories/with_both_args_and_file                                                                                                                          │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositories/with_whitespace_in_comma-separated_repos                                                                                                         │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositoriesFileFormat                                                                                                                                        │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestParseRepositoriesWithEmptyEntries                                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestValidateLabels/#00                                                                                                                                                 │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestValidateLabels/#01                                                                                                                                                 │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestValidateLabels/#02                                                                                                                                                 │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/Branch_matches_all_criteria                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/Branch_is_the_combine_branch                                                                                                                 │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestValidateLabels/#03                                                                                                                                                 │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/Branch_ends_with_the_combine_branch                                                                                                          │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/No_filters_specified                                                                                                                         │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestPrMatchesCriteriaWithMocks/Branch_and_labels_match                                                                                                                 │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/#00                                                                                                                                                    │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/--ignore-labels_match                                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestPrMatchesCriteriaWithMocks/Branch_does_not_match                                                                                                                   │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/No_filters_specified_and_partial_match_on_combine_branch_name                                                                                │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/Prefix_does_not_match                                                                                                                        │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/Suffix_does_not_match                                                                                                                        │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/Invalid_regex_pattern                                                                                                                        │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/--ignore-labels_match_(with_one_out_of_two)                                                                                                            │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/Branch_matches_prefix_only                                                                                                                   │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/Branch_matches_suffix_only                                                                                                                   │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestBranchMatchesCriteria/Branch_matches_regex_only                                                                                                                    │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestPrMatchesCriteriaWithMocks/Labels_do_not_match                                                                                                                     │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestPrMatchesCriteriaWithMocks/Neither_branch_nor_labels_match                                                                                                         │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/no_labels_match_(select_or_ignore)                                                                                                                     │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestPrMatchesCriteriaWithMocks/No_branch_or_label_filters_specified                                                                                                    │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/--select-labels_match                                                                                                                                  │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/--select-labels_match_(with_one_out_of_two)_and_ignore_labels_don't_match                                                                              │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/the_pull_request_has_no_labels                                                                                                                         │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/the_pull_request_has_no_labels_and_ignore_labels_don't_match_so_it_matches_-_but_select_labels_is_empty_so_it_means_all_labels_or_even_no_labels_match │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/the_pull_request_has_no_labels_but_we_want_to_match_the_a_label                                                                                        │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/no_label_match_criteria,_so_it_matches                                                                                                                 │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/with_one_matching_label_and_no_matching_ignore_labels_so_it_matches                                                                                    │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/the_pr_labels_match_the_select_and_ignore_labels_so_it_doesn't_match                                                                                   │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/the_pr_has_one_label_but_no_defined_ignore_or_select_labels_so_it_matches                                                                              │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/the_pr_has_one_label_and_it_is_the_select_label_so_it_matches                                                                                          │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/the_pr_has_labels_and_matching_select_labels_but_it_matches_an_ignore_label_so_it_doesn't_match                                                        │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/the_pr_has_uppercase_labels_and_we_are_using_case_insensitive_labels_so_it_matches                                                                     │ gh-combine/internal/cmd    │
│  PASS  │  0.00   │ TestLabelsMatch/the_pr_has_uppercase_labels_and_we_are_using_case_sensitive_labels_so_it_doesn't_match                                                                 │ gh-combine/internal/cmd    │
│        │         │                                                                                                                                                                        │                            │
│  PASS  │  0.00   │ TestParseRepo                                                                                                                                                          │ gh-combine/internal/github │
│  PASS  │  0.00   │ TestParseRepo/#00                                                                                                                                                      │ gh-combine/internal/github │
│  PASS  │  0.00   │ TestParseRepo/#01                                                                                                                                                      │ gh-combine/internal/github │
│  PASS  │  0.00   │ TestParseRepo/#02                                                                                                                                                      │ gh-combine/internal/github │
│  PASS  │  0.00   │ TestParseRepo/#03                                                                                                                                                      │ gh-combine/internal/github │
│  PASS  │  0.00   │ TestParseRepo/#04                                                                                                                                                      │ gh-combine/internal/github │
│        │         │                                                                                                                                                                        │                            │
╰────────┴─────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────╯
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃   FAIL  package: github.com/github/gh-combine/internal/cmd   ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

--- FAIL: TestBranchMatchesCriteria (0.00s)
--- FAIL: TestBranchMatchesCriteria/Regex_does_not_match (0.00s)

    match_criteria_test.go:280: branchMatchesCriteria("test/feature") = true; want false

╭────────┬──────────┬────────────────────────────┬───────┬──────┬──────┬──────╮
│ Status │ Elapsed  │          Package           │ Cover │ Pass │ Fail │ Skip │
├────────┼──────────┼────────────────────────────┼───────┼──────┼──────┼──────┤
│  FAIL  │  0.18s   │ gh-combine/internal/cmd    │  --   │  60  │  2   │  0   │
│  PASS  │ (cached) │ gh-combine/internal/github │  --   │  6   │  0   │  0   │
╰────────┴──────────┴────────────────────────────┴───────┴──────┴──────┴──────╯

Looking at the logs, it looks like --- FAIL: TestBranchMatchesCriteria/Regex_does_not_match (0.00s) match_criteria_test.go:280: branchMatchesCriteria("test/feature") = true; want false is the main culprit for the flakey test. Perhaps we need to take a look at the TestBranchMatchesCriteria section of test holistically to see if there is something causing these test to occasionally fail and then start passing again without any changes.

Sometimes, it even looks like multiple tests fail at once:

┃   FAIL  package: github.com/github/gh-combine/internal/cmd   ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

--- FAIL: TestBranchMatchesCriteria (0.00s)
--- FAIL: TestBranchMatchesCriteria/Branch_matches_all_criteria (0.00s)

    match_criteria_test.go:280: branchMatchesCriteria("feature/test") = false; want true

--- FAIL: TestBranchMatchesCriteria/Branch_matches_regex_only (0.00s)

    match_criteria_test.go:280: branchMatchesCriteria("feature/test") = false; want true

╭────────┬──────────┬────────────────────────────┬───────┬──────┬──────┬──────╮
│ Status │ Elapsed  │          Package           │ Cover │ Pass │ Fail │ Skip │
├────────┼──────────┼────────────────────────────┼───────┼──────┼──────┼──────┤
│  FAIL  │  0.18s   │ gh-combine/internal/cmd    │  --   │  59  │  3   │  0   │
│  PASS  │ (cached) │ gh-combine/internal/github │  --   │  6   │  0   │  0   │
╰────────┴──────────┴────────────────────────────┴───────┴──────┴──────┴──────╯

and then it will just magically resolve by re-running the tests:

╭────────┬──────────┬────────────────────────────┬───────┬──────┬──────┬──────╮
│ Status │ Elapsed  │          Package           │ Cover │ Pass │ Fail │ Skip │
├────────┼──────────┼────────────────────────────┼───────┼──────┼──────┼──────┤
│  PASS  │  0.20s   │ gh-combine/internal/cmd    │  --   │  62  │  0   │  0   │
│  PASS  │ (cached) │ gh-combine/internal/github │  --   │  6   │  0   │  0   │
╰────────┴──────────┴────────────────────────────┴───────┴──────┴──────┴──────╯
@GrantBirki GrantBirki added the bug Something isn't working label Apr 15, 2025
@GrantBirki GrantBirki self-assigned this Apr 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant