Add selector split-unit support to Test Engine Client

[meghan-kradolfer]

Description

Adds experimental selector-based split-unit support to bktec so supported non-file runners can request and execute selector plans from Test Engine. This is gated behind --selector-splitting / BUILDKITE_TEST_ENGINE_SELECTOR_SPLITTING=true, keeping existing file/example splitting behaviour unchanged by default.

Context

Linear: https://linear.app/buildkite/issue/TE-6156/add-selector-split-unit-support-to-test-engine-client

Related API/planner shape: https://linear.app/buildkite/issue/TE-6157/add-selector-split-units-to-the-test-plan-api-and-planner-model

Changes

• Adds the experimental selector splitting config field, CLI flag, and env var source.
• Adds runner capability modelling for selector splitting and marks go test as selector-capable.
• Sends selector split-unit requests for supported runners when the opt-in is enabled.
• Handles selector-format plan responses in go test execution and split summaries.
• Adds guardrail and coverage tests for the opt-in, request payloads, supported features, selector execution, and summaries.

Testing

• go test ./... — fails locally because integration-style runner tests require external binaries that are not installed in this environment (rspec, pytest, cucumber, yarn, gotestsum).

[nprizal]

Should we call this Selector?

Suggested change

	Value string `json:"value,omitempty"`
	Selector string `json:"selector,omitempty"`

[meghan-kradolfer]

The current API shape uses { "value": "github.com/..." } so I don't think we can do a full rename here but we could do:

Selector string `json:"value,omitempty"`

[meghan-kradolfer]

I’m leaning towards keeping this as value, since format: "selector" already identifies it as a selector.

[nprizal]

Ok for now. I think the problem is because we use the same type for input (API request params), output (API response), and bktec internalsw, so it is a bit confusing at the moment. Maybe we should separate them at some point. We have this top level comment for that type.

// TestCase currently can represent a single test case or a single test file (when used as output of test plan API).
// TODO: it's best if we split this into two types.

[nprizal]

Its an existing poor design of bktec, but currently we pass files to this function. However for go test, we are actually passing "packages". Maybe we should rename files to something like filesOrSelectors (Can't think of better name :D), or put a comment? Ideally we will pass selectors and the runner responsible and creating the selectors.

[meghan-kradolfer]

Updated this to be testTargets, not sure if that is any better 🤷🏼‍♀️

[nprizal]

@buildsworth-bk review

[nprizal]

Looks good to me. Will test this e2e next week before merging.

[meghan-kradolfer]

@nprizal I added this flag so this can just be merged and tested! You can see it working on this branch

It still says "no history" as we need to also turn on the TestEngineUseSelectorPrimaryTimingsForTestPlan feature flag for bk for it to hit the new selector-tag timing query

[nprizal]

I have enabled the feature flag, and it is now using the historical duration. One issue I found is that bktec sends go packages that don't have test to the Test Plan API, so the binpacking is not optimized. Packages without tests should be excluded from Test Plan API request body. I'll create a ticket for this. Meanwhile, I will merge this PR.