feat: add experiment information as OTLP telemetry attribute#29593
feat: add experiment information as OTLP telemetry attribute#29593
Conversation
- pick_experiment.cjs now writes assignments.json alongside state.json so downstream spans can read the current variant without recomputing - send_otlp_span.cjs gains readExperimentAssignments() and buildExperimentAttributes() helpers - sendJobSetupSpan and sendJobConclusionSpan both include gh-aw.experiment.<name> (one per experiment) and gh-aw.experiments (JSON of all assignments) span attributes - Tests added for pick_experiment.cjs (assignments file writing) and send_otlp_span.cjs (experiment attributes in setup and conclusion spans) Agent-Logs-Url: https://github.com/github/gh-aw/sessions/f0ab1e69-b2cd-46eb-977e-8c582c5c65a1 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…gnments.json write - buildExperimentAttributes only appends gh-aw.experiments when at least one valid variant attribute was produced - pick_experiment.cjs only writes assignments.json when at least one experiment was successfully assigned - Added test for all-empty-variant-string case in buildExperimentAttributes - Added test for <2-variant experiment skipping in pick_experiment main Agent-Logs-Url: https://github.com/github/gh-aw/sessions/f0ab1e69-b2cd-46eb-977e-8c582c5c65a1 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds A/B experiment assignment metadata to OTLP spans so observability backends can correlate traces with experiment variants.
Changes:
- Persist current-run experiment assignments to an
assignments.jsonfile inpick_experiment.cjs. - Read assignments in
send_otlp_span.cjsand emit them as OTLP span attributes (gh-aw.experiment.<name>andgh-aw.experiments). - Add/extend test coverage for both the assignments file behavior and OTLP attribute emission.
Show a summary per file
| File | Description |
|---|---|
| actions/setup/js/pick_experiment.cjs | Writes current-run experiment assignments to assignments.json when at least one assignment exists. |
| actions/setup/js/pick_experiment.test.cjs | Adds tests validating assignments.json creation/overwrite and no-write scenarios. |
| actions/setup/js/send_otlp_span.cjs | Reads experiment assignments and injects them into setup/conclusion OTLP spans; exports new helpers/constants. |
| actions/setup/js/send_otlp_span.test.cjs | Adds tests for reading/building experiment attributes and ensuring spans include/omit them appropriately. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 4/4 changed files
- Comments generated: 2
| /** | ||
| * Path to the experiment assignments file written by pick_experiment.cjs. | ||
| * Contains a JSON object mapping experiment name → selected variant for the | ||
| * current workflow run. Example: `{"caveman":"yes","style":"detailed"}`. | ||
| * @type {string} | ||
| */ | ||
| const EXPERIMENT_ASSIGNMENTS_PATH = "/tmp/gh-aw/experiments/assignments.json"; |
| function buildExperimentAttributes(assignments) { | ||
| if (!assignments || typeof assignments !== "object") return []; | ||
| const names = Object.keys(assignments).sort(); | ||
| if (names.length === 0) return []; | ||
| const attrs = []; | ||
| for (const name of names) { | ||
| const variant = assignments[name]; | ||
| if (typeof variant === "string" && variant) { | ||
| attrs.push(buildAttr(`gh-aw.experiment.${name}`, variant)); | ||
| } | ||
| } | ||
| if (attrs.length > 0) { | ||
| attrs.push(buildAttr("gh-aw.experiments", JSON.stringify(assignments))); | ||
| } | ||
| return attrs; |
🧪 Test Quality Sentinel ReportTest Quality Score: 83/100✅ Excellent
Test Classification DetailsView all 17 test classifications
Score Breakdown
The only deduction is the test inflation flag on Language SupportTests analyzed:
No Go test files were added or modified in this PR. Notes on Mocking StrategyThe tests make heavy use of Verdict
📖 Understanding Test ClassificationsDesign Tests (High Value) verify what the system does:
Implementation Tests (Low Value) verify how the system does it:
Goal: Shift toward tests that describe the system's behavioral contract — the promises it makes to its users and collaborators. References: §25228044513
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
@copilot review all comments, fix https://github.com/github/gh-aw/actions/runs/25228047833/job/73976430444?pr=29593 |
…tered experiments JSON - readExperimentAssignments() now derives the file path from GH_AW_EXPERIMENT_STATE_DIR env var (same as pick_experiment.cjs), falling back to EXPERIMENT_ASSIGNMENTS_PATH when not set - buildExperimentAttributes() builds gh-aw.experiments JSON from the filtered valid assignments only (excludes empty/non-string variants), making it consistent with the per-experiment attributes - Updated/added tests for both fixes Agent-Logs-Url: https://github.com/github/gh-aw/sessions/5ccb46be-572d-4706-9cd6-c6f4ee50a256 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Fixed both review comments in ad409cb:
The CI failure in |
Summary
Adds experiment assignment information as attributes in the OTLP OpenTelemetry telemetry spans, enabling A/B experiment variant correlation in observability backends.
Changes
actions/setup/js/pick_experiment.cjs/tmp/gh-aw/experiments/assignments.jsonalongside the existing state file (which tracks counts)actions/setup/js/send_otlp_span.cjsEXPERIMENT_ASSIGNMENTS_PATHconstant (/tmp/gh-aw/experiments/assignments.json)readExperimentAssignments()helper that reads the assignments file (returnsnullif absent/invalid — non-fatal)buildExperimentAttributes()helper that converts assignments to OTLP span attributes:gh-aw.experiment.<name>attribute per experiment (e.g.gh-aw.experiment.caveman=yes)gh-aw.experimentsJSON string attribute (only when at least one valid variant exists)sendJobSetupSpanandsendJobConclusionSpannow include experiment attributesTests
pick_experiment.test.cjs: 3 new tests covering assignments file creation, successive-run overwrite, and no-write casessend_otlp_span.test.cjs: 10 new tests coveringreadExperimentAssignments,buildExperimentAttributes, and experiment attribute inclusion in both setup and conclusion spansAttribute design
gh-aw.experiment.<name>gh-aw.experiment.caveman=yesgh-aw.experiments{"caveman":"yes","style":"detailed"}The per-experiment attributes allow filtering on individual variants in dashboards. The aggregated JSON attribute enables substring search in backends that don't support per-attribute filtering.