init: import whichtests sources#1
Conversation
Imports the testselect Go test-plan generator from coder/coder@bf6a0d953f (branch ethan/go-test-flake-detector) and splits the single ~1.6k-line main.go into ten focused files inside package main: - cli.go: entrypoint, flag parsing, command orchestration - config.go: config / commandConfig types and defaults - request.go: runRequest, diffRange, revision validation - gitexec.go: gitRunner / gitFetcher types and exec.Command impl - diff.go: git diff parsing, change kinds, hunks, line ranges - snapshot.go: AST snapshot parsing, fileSnapshot, sharedDecl, fallbacks - broadening.go: per-kind broadening rules (broadeningScope) - selection.go: per-change selection logic - inventory.go: inventoryCache for package/directory test discovery - plan.go: plan construction, matrix and summary rendering githubactions.go and publish.go are imported unchanged. Tests (githubactions_test.go, integration_test.go, main_test.go) are kept as single files initially. go vet, go build, go test, and go test -race are all green.
Split main_test.go (2297 lines) into focused *_test.go files mirroring the source layout, and move two helpers (mergePackageSelection, packagePattern) from plan.go to selection.go so layering is consistent (plan depends on selection, not vice versa). Test files now line up 1:1 with their production counterparts: cli_test.go - run/runCommand end-to-end tests diff_test.go - hunk/change parsing tests snapshot_test.go - parseFileSnapshot/fallback tests selection_test.go - selectTestsForSnapshots and merge tests plan_test.go - buildExecutionPlan/renderSummary tests publish_test.go - publishPlan tests (from githubactions_test.go) githubactions_test.go - GitHub event/range tests integration_test.go - real-git integration (unchanged) helpers_test.go - shared line-range + inventory helpers gitfake_test.go - fakeGitRepo mock + dispatcher methods go vet, go build, go test, and go test -race all pass.
Mirrors the parallel testctx repo's structure: Makefile, golangci-lint v2 config, prettier config, gitignore, MIT license, CI workflow with fmt/lint/test jobs, and the composite setup-go action (default Go version bumped to 1.26.2 to match go.mod). Drive-by gofumpt fix: blank line between needsOldSnapshot and addMatchingTests in selection.go.
- publish.go: surface Close errors via named return on appendFile, fixing errcheck. Close errors on a write path can indicate lost data, so propagating beats silently discarding. - plan.go: drop `:=` in selectTestPlan loop so the inner err reuses the outer binding, fixing govet shadow. - githubactions.go: drop the redundant cfg.config selector and rely on the promoted method, fixing staticcheck QF1008. - helpers_test.go: drop both dir and packageName parameters from mustPackageInventory; all 16 call sites pass the same synthetic values, so hardcode them and document the helper, fixing unparam. go vet, go build, make test, and make fmt are clean. make lint still reports six gosec findings (G301/G304) covering directory permissions and reads from user-provided paths; those are policy decisions.
The previous name was honest but generic — many things 'select tests.' 'whichtests' is the question this tool answers: given a Go PR's diff, which tests should run? Renaming makes the intent obvious in workflow YAML, CLI help, and the module path. Touchpoints: - go.mod module github.com/coder/whichtests - Makefile builds build/whichtests - cli.go doc comment - README.md heading and invocation examples Internal symbol names like selectTestsForSnapshots are untouched — they describe what the function does, not the tool's identity.
ethanndickson
left a comment
There was a problem hiding this comment.
Thanks for extracting this from coder/coder. The algorithm is small, the git/AST/matrix boundaries are kept clean, and the regex gates on safeTestNameRE/safePackagePatternRE plus the GITHUB_OUTPUT CRLF + 1 MiB guard close the highest-impact injection paths. The two real-git integration tests are a particularly strong asset.
Reviewer context: this binary has exactly one consumer — coder/coder's flake-go.yaml, invoked as whichtests --repo-root . --github-actions --out-matrix "$RUNNER_TEMP/flake-matrix.json" after a per-PR go install from a pinned SHA (no binary caching). Severity is calibrated to that flow.
11 reviewers in parallel cross-checked into 6 P2, 8 P3, 1 perf bundle, 2 nit bundles across 17 inline comments. Most P2s are deletions (dead config, fossil fields, a test-only run() parallel entry that 9 reviewers independently flagged) plus two algorithm findings the smoke test exposed. None block merge.
| MergeBaseRef string | ||
| Sinks outputSinks | ||
| OutputSizeLimit int | ||
| } |
There was a problem hiding this comment.
P2 runRequest.OutputSizeLimit is plumbed through but never written. (Structural Analyst P2, Edge Case Analyst P2, Go Architect P2, Product Reviewer P3)
OutputSizeLimitis declared onrunRequest, threaded throughpublishPlan(... outputSizeLimit int), threaded again intoappendGitHubOutput(... outputSizeLimit int), and finally compared insideappendGitHubOutputwith a 0-means-default escape (if outputSizeLimit == 0 { outputSizeLimit = defaultGitHubOutputValueLimit }).rgfinds zero writers ofreq.OutputSizeLimitanywhere in production or tests; both invocation sites pass 0.
Delete the field, drop the parameter from publishPlan/appendGitHubOutput, and reference defaultGitHubOutputValueLimit directly inside appendGitHubOutput. The existing publish_test.go tests pass the limit explicitly to appendGitHubOutput, so they keep their existing signatures — the deletion only touches the runRequest → publishPlan plumbing. Three layers of conditional plumbing carry one constant; this is the kind of "looks load-bearing, isn't" hook that traps future maintainers.
| } | ||
|
|
||
| type testDecl struct { | ||
| FilePath string |
There was a problem hiding this comment.
P2 testDecl{FilePath, Range} is a set in disguise — both fields are written, neither is read. (Structural Analyst P2, Go Architect P2)
the type is
map[string][]testDeclandtestDecl{FilePath, Range}is written ininventoryCache.loadPackageInventoryand never read. The only readers arehasTest(name)(presence check on the map key) andallTests()(slices.Sorted(maps.Keys(...))). The slice value, including bothtestDeclfields, is dead.rg -n '\.FilePath�'andrg -n 'testDecl�'confirm no field reads outside the constructors.
Collapse Tests map[string][]testDecl → Tests map[string]struct{} and delete testDecl. The hasTest/allTests accessors stay identical (they already only use keys), and mustPackageInventory in helpers_test.go simplifies. The struct currently misleads readers into believing the inventory layer knows where each test lives — it doesn't.
| func (decl sharedDecl) broadeningScopeOnNewSide(oldSnapshot *fileSnapshot) broadeningScope { | ||
| switch decl.Kind { | ||
| case sharedDeclImport: | ||
| return broadeningPackage |
There was a problem hiding this comment.
P2 Package-wide broadening on any new import is the dominant degenerate matrix in the smoke test — selectivity is not earning its keep. (Product Reviewer P2)
per context.md's smoke-test note, roughly half of the non-empty matrices were driven by a single stdlib
importaddition, each then hitting themaxBroadenedTests = 50cap and falling back toRunAll = truewithtest_count = 1. End state: the matrix entry runs every test in the touched package exactly once, with no shuffle multiplier. That is approximately the same coverage flake-go would produce by running the touched package unconditionally, so the selector's selectivity is not earning its keep on these PRs.
Two cheap fixes worth considering:
- Treat stdlib imports as non-broadening — they cannot introduce init-time side effects via a new module, so the touched test func plus its in-file siblings is sufficient.
- When the broadened set blows past
maxBroadenedTests, prefer re-narrowing to the tests literally touched in the hunk over the current "drop to RunAll, demote totest_count = 1" path. The current fallback inverts the product value — flake reruns are most useful attest_count = 10, but this is exactly the case where the binary downgrades them to 1×.
Happy to scope this to a follow-up if you'd rather keep this PR a clean lift.
| ) | ||
|
|
||
| func broadeningScopeForOldHunk(decls []sharedDecl, candidate lineRange) broadeningScope { | ||
| for _, decl := range decls { |
There was a problem hiding this comment.
P2 broadeningScopeFor{Old,New}Hunk returns the first overlapping decl's scope rather than the highest; a hunk spanning an Import and an Init under-broadens (Package instead of Directory). (Edge Case Analyst P2)
broadeningScopeForOldHunkiteratesdeclsin source order and returns immediately on first overlap (regardless of whether the result isbroadeningPackageor higher).broadeningScopeForNewHunkis slightly better — it skipsbroadeningNone— but still returns the first non-None match. Go files typically have imports (Package) → vars/types → funcs →init/TestMainin source order, so a hunk that overlaps an Import and an Init returnsbroadeningPackageand misses the directory-wide broadening that should apply.
--unified=0is used (diff.go:196), so hunks are tight to changed lines and overlapping two decls requires a contiguous edit across two adjacent decls — uncommon but real. Example: adding a single line between the lastimportblock andfunc init().
Fix: scan all overlapping decls and return max(scope). Cheap, local, and removes a silent correctness gap. A regression test with the import→init adjacency would pin this down.
| } | ||
|
|
||
| func run(ctx context.Context, cfg config, stdout, stderr io.Writer, git gitRunner) error { | ||
| req, err := explicitRunRequest(cfg) |
There was a problem hiding this comment.
P2 The test-only run() shim is a parallel composition path; runCommand (production) has no end-to-end test, so regressions in fetch wiring or sink wiring would slip past the suite. (Structural Analyst P2, Test Auditor P3, Contract Auditor P3, Edge Case Analyst P3, Performance Analyst P3, Product Reviewer P3, Duplication Checker P3, Go Architect P3, Style Reviewer Nit)
maincallsrunCommand(...). The thin wrapperrun(explicitRunRequest+executeRunRequestwithfetch=nil) is the entry every test incli_test.goandintegration_test.gouses;rg -n 'runCommand\('finds exactly one caller (main). Tests ofgithubActionsRunRequeststop at therunRequestvalue — they never feed it throughexecuteRunRequest/publishPlan. So the only thing that pins the production composition together (theif cfg.GitHubActionsbranch, the call toexecuteRunRequest, theos.Stdout/os.Stderrwiring, the size-limit defaulting viareq.OutputSizeLimit=0) ismain, which is never executed in tests.
This is the most-convergent finding in the review — 9 of 11 reviewers flagged it independently. Concrete fix: delete run, migrate the 19 test sites to runCommand(ctx, commandConfig{config: cfg}, ..., nil), and add the missing TestRunCommandGitHubActionsEnd2End that wires runCommand against a fake event payload (Test Auditor specifically called this out as the highest-value glue test currently missing). The deletion + new test collapses both code paths into one and adds end-to-end coverage of the --github-actions path that today only the live consumer exercises.
| result.Matrix.Include = result.Matrix.Include[:keep] | ||
| result.Matrix.Include = append(result.Matrix.Include, matrixEntry{ | ||
| Package: strings.Join(overflowPackages, " "), | ||
| TestCount: runOnceTargetCount, |
There was a problem hiding this comment.
P3 Matrix overflow's Package field is a space-joined argv-list, not a single path; the contract is implicit at three boundaries and undocumented. (Contract Auditor Obs, Edge Case Analyst P3, Structural Analyst Obs, Go Architect P4)
overflowPackages := strings.Join(overflowPackages, " ")produces a string like"./a ./b ./c". This works because flake-go.yaml will spread it intogo test ${{matrix.package}}, where YAML→bash word-splitting expands it into multiple args. If a future consumer ever wraps the value in quotes ("${{matrix.package}}") or maps it to a${{matrix.package}}field that doesn't undergo splitting (e.g. anactions/upload-artifactname), the overflow target will fail in a confusing way.
I traced the actual path on the coder/coder side to verify: test-go-pg/action.yaml:104 sets TEST_PACKAGES: ${{ inputs.test-packages }} (env var, preserves spaces); Makefile:1470 runs gotestsum --packages="$(TEST_PACKAGES)"; gotestsum itself splits --packages on whitespace. So the contract works today, but the chain is "Go binary emits space-joined string → YAML env var → Make recipe with quoted shell expansion → gotestsum's internal whitespace split" — four implicit assumptions in a row.
Two cleaner shapes: (a) emit packages []string as a JSON array and have the workflow join with ${{ join(matrix.packages, ' ') }}; (b) keep the current shape but add a one-line code comment at line 159 documenting that Package is an argv-list when Broadened && len(targets) > maxMatrixEntries so a future maintainer doesn't "fix" it by quoting on either side.
| } | ||
|
|
||
| func validateRevision(flagName, revision string) error { | ||
| if revision == "" { |
There was a problem hiding this comment.
P3 validateRevision is much looser than the GitHub-supplied SHAs it protects, and the fields it validates are named BaseSHA/HeadSHA — the name promises something the validator does not enforce. (Security Reviewer P3, Contract Auditor P3)
the only checks are non-empty, no leading
-, no:, no NUL. Thepull_request.base.sha,pull_request.head.sha,workflow_dispatch.inputs.*all flow through this validator before being concatenated intorevision + ":" + filePath(diff.go:270),revision + "^{commit}"(gitexec.go:25), and positional args togit merge-base,git diff,git ls-tree. No command injection is reachable — arguments don't pass through a shell, and the leading--gate prevents flag confusion. But the symmetry withvalidateRef(which forbids@{,..,*, control/whitespace, leading.) is missing for what should be the most rigidly-typed input — a 40-char hex SHA.
Contract Auditor adds: the README's local-debug example uses
--base-sha origin/main, so the field happily accepts arbitrary revspecs. The promise from the name ("SHA") is broken by the validator and contradicted by the documentation.
Two equivalent fixes: tighten validateRevision to ^[0-9a-fA-F]{7,64}$ on the GitHub Actions path (where the input is from GitHub-set payload), or rename BaseSHA/HeadSHA → BaseRev/HeadRev (and --base-sha/--head-sha → --base-rev/--head-rev) and document that revspecs are accepted. The README example would have to update either way.
| } | ||
| for testName, declRange := range snapshot.tests { | ||
| inventory.Tests[testName] = append(inventory.Tests[testName], testDecl{FilePath: filePath, Range: declRange}) | ||
| } |
Initial import of the
whichteststool, extracted fromcoder/coder'sscripts/testselect/and renamed.whichtestsanswers: given a PR diff, which Go tests should run? The flake-go workflow uses its output to rerun new/modified tests under shuffle and surface flakes early.Commits
init: importscripts/testselectfromcoder/coderand splitmain.goacross cohesive files (cli, config, request, gitexec, diff, snapshot, broadening, selection, inventory, plan, publish, githubactions)split: organize tests along source-file boundaries (1:1 with sources)chore: import boilerplate fromcoder/paralleltestctx(.gitignore, .golangci.yaml, .prettierrc, LICENSE, Makefile, CI workflow, setup-go composite action)fix: address mechanical lint findings (errcheck, govet shadow, staticcheck QF1008, unparam)rename: testselect -> whichtests (module path, Makefile target, CLI doc comment, README)Outstanding
gosecfindings (G304/G301) deferred for review. Recommendation is to lowerpublish.goMkdirAll perms to0o750and#nosecannotate the remaining 4 sites (test temp dir, GITHUB_EVENT_PATH, --out-matrix and --out-summary flag values).coder/coder'sscripts/testselect/directory and theflake-go.yamlreference still need to be migrated to use this repo; that's out of scope for this PR.Relates to CODAGT-381