internal: [TE-5828] Add bktec plan step to collect commit metadata on agent builds#3924
Merged
Merged
Conversation
Adds a discardable plan-step script for the Test Engine plan metadata
side-channel. Script downloads bktec v2.5.0 from the test-engine-client
GitHub release at step time, then runs:
BKTEC_PREVIEW_SELECTION=1 bktec plan --json --collect-git-metadata > /dev/null
Plan output is discarded -- the goal is the metadata write, not the plan.
Sets MAX_PARALLELISM=2 / TARGET_TIME=1m so the Test Engine API accepts
the plan request (it rejects parallelism = 0 even on metadata-only
requests; surfaced by TE-5766 verification on bk/bk-rspec).
bktec is installed via curl rather than bumping go.mod because the agent
repo pins test-engine-client v1.6.0 for use as 'go tool test-engine-client'
on the runtime test path. v1.6.0 predates the 'plan' subcommand (v2.0.0)
and the --collect-git-metadata flag (v2.4.0). A coordinated bump is left
as follow-up.
Adds a top-level 🧪 Collect commit metadata step in .buildkite/pipeline.yml between the lint group and the Tests group. Step characteristics: - soft_fail: true -- a Test Engine API hiccup never blocks the build. - No depends_on, no dependents -- runs in parallel with everything from t=0; nothing waits on it. - Top-level placement (not inside the Tests group) so metadata is collected on every build, including doc-only PRs that don't trigger the tests group's if_changed. - Bare-agent step (no docker-compose) -- .git is present from the agent's normal checkout, so no app-with-git-style compose service is needed (unlike bk/bk-rspec in TE-5640). Sets BUILDKITE_TEST_ENGINE_SUITE_SLUG=buildkite-agent and BUILDKITE_TEST_ENGINE_TEST_RUNNER=gotest at the step level for consistency with .buildkite/steps/tests.sh.
gchan
changed the title
gchan
changed the title
First verification build (12477) failed with:
failed to run go list: exec: "go": executable file not found in $PATH
bktec's gotest runner shells out to `go list ./...` for package
discovery (see test-engine-client/internal/runner/gotest.go GetFiles).
The bare agent runner has no Go on PATH; only the `agent` docker-compose
service does (it builds from Dockerfile-compile, which is based on
golang:1.25.10).
Wraps the step in the existing `agent` docker-compose service via the
docker-compose plugin, matching how .buildkite/steps/tests.sh is wired.
Side benefit: BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN is already declared
on this service in .buildkite/docker-compose.yml, removing the
bare-agent token-availability unknown flagged in the PR description.
propagate-environment: true forwards the step-level
BUILDKITE_TEST_ENGINE_* env vars into the container.
Step now runs inside the `agent` docker-compose service which is
linux-only (golang:1.25.10 base image). Drop OS detection.
Drop the ${VAR:-default} fallbacks for BUILDKITE_TEST_ENGINE_* env vars
-- they're all set at the pipeline step level in pipeline.yml, so the
script-level defaults were belt-and-braces.
Fold the multi-line curl and bktec invocations onto single lines.
83 -> 58 lines, no behaviour change. Arch detection stays -- the
compose service still runs on both amd64 and arm64 runners.
Contributor
Author
|
Verified metadata is being collected. |
buildkate
approved these changes
May 14, 2026
Contributor
buildkate
left a comment
buildkate
left a comment
There was a problem hiding this comment.
LGMT. Will the be reverted once the model has been trained or is this a permanent fixture?
Contributor
Author
Likely permanent-ish as we would like to collect new data for periodic re-training. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a discardable
bktec plan --collect-git-metadatastep to theagent build pipeline so git commit metadata is sent to Buildkite Test
Engine on every build. The plan output is discarded -- existing test
steps continue to use their declared
parallelism:values, so this isa metadata-only side-channel with no run-time contract change.
bktecv2.5.0 is downloaded from thetest-engine-clientGitHubrelease at step time. The agent repo pins
test-engine-client v1.6.0in
go.modfor use asgo tool test-engine-clientin.buildkite/steps/tests.sh, which predates theplansubcommand(v2.0.0) and the
--collect-git-metadataflag (v2.4.0). A coordinatedbump of both call sites is tracked as follow-up.
The step is
soft_fail: true, has nodepends_onand no dependents,so a Test Engine API hiccup never blocks the build. It runs inside the
existing
agentdocker-compose service to pick upgoon PATH (bktecshells out to
go list ./...for package discovery) and theBUILDKITE_TEST_ENGINE_API_ACCESS_TOKENenv var.Context
Internal ticket: TE-5828.
Changes
Best reviewed commit-by-commit:
Verification
The new
:test_tube: Collect commit metadatastep should:bktec 2.5.0after the download.Plan request issued -- git commit metadata sent to Test Engine.AI assisted.
Deployment
Low risk. New step is
soft_fail: trueand has no dependents.Rollback
Yes -- single revert.