fix a flakey test #22
Description
Occasionally, it looks like the following test fails:
$ script/test
╭────────┬─────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────────────────────────╮
│ Status │ Elapsed │ Test │ Package │
├────────┼─────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────────────────────────┤
│ PASS │ 0.01 │ TestParseRepositories │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestCreatePullRequest │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestValidateLabels │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestPrMatchesCriteriaWithMocks │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestDisplayTableStats │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestDisplayJSONStats │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestDisplayPlainStats │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestApplyDefaultOwner │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestApplyDefaultOwner/empty_default_owner │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestApplyDefaultOwner/repo_already_has_owner │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestApplyDefaultOwner/repo_needs_default_owner │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositories/empty_inputs │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositories/single_repository_arg │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositories/multiple_repository_args │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositories/comma-separated_repository_args │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositories/with_default_owner │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositories/with_valid_file │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositories/with_invalid_file │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositories/with_both_args_and_file │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositories/with_whitespace_in_comma-separated_repos │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositoriesFileFormat │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestParseRepositoriesWithEmptyEntries │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestValidateLabels/#00 │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestValidateLabels/#01 │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestValidateLabels/#02 │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/Branch_matches_all_criteria │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/Branch_is_the_combine_branch │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestValidateLabels/#03 │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/Branch_ends_with_the_combine_branch │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/No_filters_specified │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestPrMatchesCriteriaWithMocks/Branch_and_labels_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/#00 │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/--ignore-labels_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestPrMatchesCriteriaWithMocks/Branch_does_not_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/No_filters_specified_and_partial_match_on_combine_branch_name │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/Prefix_does_not_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/Suffix_does_not_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/Invalid_regex_pattern │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/--ignore-labels_match_(with_one_out_of_two) │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/Branch_matches_prefix_only │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/Branch_matches_suffix_only │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestBranchMatchesCriteria/Branch_matches_regex_only │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestPrMatchesCriteriaWithMocks/Labels_do_not_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestPrMatchesCriteriaWithMocks/Neither_branch_nor_labels_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/no_labels_match_(select_or_ignore) │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestPrMatchesCriteriaWithMocks/No_branch_or_label_filters_specified │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/--select-labels_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/--select-labels_match_(with_one_out_of_two)_and_ignore_labels_don't_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/the_pull_request_has_no_labels │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/the_pull_request_has_no_labels_and_ignore_labels_don't_match_so_it_matches_-_but_select_labels_is_empty_so_it_means_all_labels_or_even_no_labels_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/the_pull_request_has_no_labels_but_we_want_to_match_the_a_label │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/no_label_match_criteria,_so_it_matches │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/with_one_matching_label_and_no_matching_ignore_labels_so_it_matches │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/the_pr_labels_match_the_select_and_ignore_labels_so_it_doesn't_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/the_pr_has_one_label_but_no_defined_ignore_or_select_labels_so_it_matches │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/the_pr_has_one_label_and_it_is_the_select_label_so_it_matches │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/the_pr_has_labels_and_matching_select_labels_but_it_matches_an_ignore_label_so_it_doesn't_match │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/the_pr_has_uppercase_labels_and_we_are_using_case_insensitive_labels_so_it_matches │ gh-combine/internal/cmd │
│ PASS │ 0.00 │ TestLabelsMatch/the_pr_has_uppercase_labels_and_we_are_using_case_sensitive_labels_so_it_doesn't_match │ gh-combine/internal/cmd │
│ │ │ │ │
│ PASS │ 0.00 │ TestParseRepo │ gh-combine/internal/github │
│ PASS │ 0.00 │ TestParseRepo/#00 │ gh-combine/internal/github │
│ PASS │ 0.00 │ TestParseRepo/#01 │ gh-combine/internal/github │
│ PASS │ 0.00 │ TestParseRepo/#02 │ gh-combine/internal/github │
│ PASS │ 0.00 │ TestParseRepo/#03 │ gh-combine/internal/github │
│ PASS │ 0.00 │ TestParseRepo/#04 │ gh-combine/internal/github │
│ │ │ │ │
╰────────┴─────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────╯
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ FAIL package: github.com/github/gh-combine/internal/cmd ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
--- FAIL: TestBranchMatchesCriteria (0.00s)
--- FAIL: TestBranchMatchesCriteria/Regex_does_not_match (0.00s)
match_criteria_test.go:280: branchMatchesCriteria("test/feature") = true; want false
╭────────┬──────────┬────────────────────────────┬───────┬──────┬──────┬──────╮
│ Status │ Elapsed │ Package │ Cover │ Pass │ Fail │ Skip │
├────────┼──────────┼────────────────────────────┼───────┼──────┼──────┼──────┤
│ FAIL │ 0.18s │ gh-combine/internal/cmd │ -- │ 60 │ 2 │ 0 │
│ PASS │ (cached) │ gh-combine/internal/github │ -- │ 6 │ 0 │ 0 │
╰────────┴──────────┴────────────────────────────┴───────┴──────┴──────┴──────╯Looking at the logs, it looks like --- FAIL: TestBranchMatchesCriteria/Regex_does_not_match (0.00s) match_criteria_test.go:280: branchMatchesCriteria("test/feature") = true; want false is the main culprit for the flakey test. Perhaps we need to take a look at the TestBranchMatchesCriteria section of test holistically to see if there is something causing these test to occasionally fail and then start passing again without any changes.
Sometimes, it even looks like multiple tests fail at once:
┃ FAIL package: github.com/github/gh-combine/internal/cmd ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
--- FAIL: TestBranchMatchesCriteria (0.00s)
--- FAIL: TestBranchMatchesCriteria/Branch_matches_all_criteria (0.00s)
match_criteria_test.go:280: branchMatchesCriteria("feature/test") = false; want true
--- FAIL: TestBranchMatchesCriteria/Branch_matches_regex_only (0.00s)
match_criteria_test.go:280: branchMatchesCriteria("feature/test") = false; want true
╭────────┬──────────┬────────────────────────────┬───────┬──────┬──────┬──────╮
│ Status │ Elapsed │ Package │ Cover │ Pass │ Fail │ Skip │
├────────┼──────────┼────────────────────────────┼───────┼──────┼──────┼──────┤
│ FAIL │ 0.18s │ gh-combine/internal/cmd │ -- │ 59 │ 3 │ 0 │
│ PASS │ (cached) │ gh-combine/internal/github │ -- │ 6 │ 0 │ 0 │
╰────────┴──────────┴────────────────────────────┴───────┴──────┴──────┴──────╯
and then it will just magically resolve by re-running the tests:
╭────────┬──────────┬────────────────────────────┬───────┬──────┬──────┬──────╮
│ Status │ Elapsed │ Package │ Cover │ Pass │ Fail │ Skip │
├────────┼──────────┼────────────────────────────┼───────┼──────┼──────┼──────┤
│ PASS │ 0.20s │ gh-combine/internal/cmd │ -- │ 62 │ 0 │ 0 │
│ PASS │ (cached) │ gh-combine/internal/github │ -- │ 6 │ 0 │ 0 │
╰────────┴──────────┴────────────────────────────┴───────┴──────┴──────┴──────╯