[ab-advisor] A/B experiment: prompt_style for daily-community-attribution by Copilot · Pull Request #29620 · github/gh-aw

Copilot · 2026-05-01T20:30:02Z

Implements the prompt_style experiment campaign on daily-community-attribution to test whether a concise prompt reduces token consumption (≥20% target) without degrading attribution quality.

Changes

Frontmatter: adds experiments: prompt_style: [concise, verbose] — round-robin variant selection, ~20 runs per variant over ~40 days
Sections 1–6 + Token Budget Guidelines: each wrapped with {{#if experiments.prompt_style == "concise"}} / {{#else}} / {{#endif}} blocks
Concise variants: strip narration, formatting examples, and redundant reminders; retain all operational constraints (file paths, tier caps, size limits, deferred-processing rules, read-before-write instructions)
Lock file: regenerated via gh aw compile
Template engine: extended to support {{#else}} / {{#endif}} syntax and GitHub Actions script style equality expressions (v == "b")

experiments:
  prompt_style: [concise, verbose]

{{#if experiments.prompt_style == "concise"}}
### 1. Attribute Issues

Read `pre_attributed.json` (Tier 0–2, pre-computed — do not re-derive). For each
entry in `tier3_candidates_capped.json` (≤5), apply Tier 3 (one `issue_read` call per
issue). Anything unresolved → Tier 4. Issues beyond the first 5 in
`tier3_candidates.json` are deferred to the next run — do not process them.
{{#else}}
### 1. Attribute All Resolved Community Issues
...full verbose instructions...
{{#endif}}

Template Engine Additions (`actions/setup/js`)

is_truthy.cjs: evaluates GitHub Actions script style equality expressions — concise == "concise" → true; verbose == "concise" → false; also supports ===, !=, !==
interpolate_prompt.cjs step 2.5: substitutes experiments.NAME references inside {{#if ...}} conditions so the condition resolves to the actual runtime variant value before template rendering
interpolate_prompt.cjs renderMarkdownTemplate: added {{#else}} branch splitting and {{#endif}} as primary closing tag (with {{/if}} retained as alternate) — keeps the true branch when condition is truthy, the false branch when falsy
Tests: added coverage for v == b equality in is_truthy.test.cjs and for {{#else}} / {{#endif}} rendering in interpolate_prompt.test.cjs

…ution workflow - Add `experiments: prompt_style: [concise, verbose]` to frontmatter - Wrap Sections 1-6 and Token Budget Guidelines with concise/verbose conditional blocks using `{{#if (eq experiments.prompt_style "concise")}}` - Regenerate lock file via `gh aw compile` Closes #<issue> Agent-Logs-Url: https://github.com/github/gh-aw/sessions/fc6233bc-38b0-43b7-9e11-4a0a345419b3 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

…tructions - Section 1: add deferred-processing note (issues beyond 5 deferred to next run) - Section 2: add explicit "read the existing wiki page" before merge instruction Agent-Logs-Url: https://github.com/github/gh-aw/sessions/fc6233bc-38b0-43b7-9e11-4a0a345419b3 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-05-01T20:46:45Z


 ## Workflow

+{{#if (eq experiments.prompt_style "concise")}}


Validate that this macro syntax is correct and recognized by action macros.

Validated — the {{#if (eq experiments.prompt_style "concise")}} syntax was not previously recognized. The isTruthy function only handled simple falsy strings, so the (eq ...) expression was always truthy, and the condition never switched variants.

Fixed in commit 3aac568:

is_truthy.cjs now evaluates (eq VALUE "LITERAL") expressions correctly

interpolate_prompt.cjs step 2.5 now substitutes experiments.NAME references inside {{#if ...}} conditions (in addition to __PLACEHOLDER__ forms), so by the time the template renderer runs, the condition is (eq concise "concise") or (eq verbose "concise") with the actual variant value in place

pelikhan · 2026-05-01T20:47:07Z

+entry in `tier3_candidates_capped.json` (≤5), apply Tier 3 (one `issue_read` call per
+issue). Anything unresolved → Tier 4. Issues beyond the first 5 in
+`tier3_candidates.json` are deferred to the next run — do not process them.
+{{else}}


@copilot, validate that this syntax is correct.

Validated — {{else}} was not supported. The template renderer (renderMarkdownTemplate) had no {{else}} handling, so when the condition was truthy the entire body (including {{else}} text and the verbose branch) was kept as literal output. Both branches would appear in the prompt, and {{else}} would show up as raw text.

Fixed in commit 3aac568: renderMarkdownTemplate now splits the block body on {{else}} and keeps only the appropriate branch based on the condition's truthiness.

- is_truthy.cjs: handle (eq VALUE "LITERAL") equality helper expressions so {{#if (eq concise "concise")}} evaluates correctly after experiment placeholder substitution - interpolate_prompt.cjs: split {{#if}}...{{else}}...{{/if}} blocks on the {{else}} separator and keep the appropriate branch based on truthiness - interpolate_prompt.cjs step 2.5: substitute experiments.NAME references inside {{#if ...}} conditions (in addition to __PLACEHOLDER__ forms) so (eq experiments.prompt_style "value") conditions resolve to actual values - is_truthy.test.cjs: add tests for (eq ...) helper - interpolate_prompt.test.cjs: add tests for {{else}} and (eq ...) rendering Agent-Logs-Url: https://github.com/github/gh-aw/sessions/ae04afdf-c413-4a21-bbbe-1632b9f369f0 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-05-01T21:08:19Z

@copilot

the {{else}} syntax should be {{#else}}
the closing of {{/if}} syntax should be {{#endif}}. Alternate {{/if}} are also supported
do not add support for eq v b operator, assume use GitHub actions script style v == b

…edback - daily-community-attribution.md: {{else}} → {{#else}}, {{/if}} → {{#endif}}, (eq experiments.prompt_style "concise") → experiments.prompt_style == "concise" - is_truthy.cjs: replace (eq v b) helper with GitHub Actions script style v == "b" / v === "b" / v != "b" / v !== "b" equality expressions - interpolate_prompt.cjs: {{else}} split → {{#else}}; closing tag regex now matches {{#endif}} (primary) or {{/if}} (alternate) in both passes; update comments - is_truthy.test.cjs: replace (eq ...) tests with v == b style tests - interpolate_prompt.test.cjs: update new tests to use {{#else}}/{{#endif}}/v==b; add test verifying {{/if}} alternate closing tag still works Agent-Logs-Url: https://github.com/github/gh-aw/sessions/be990686-bfaf-4163-8e65-1b4c281f0a98 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-05-01T21:15:44Z

@copilot

the {{else}} syntax should be {{#else}}...

Done in commit 6502dc5:

{{else}} → {{#else}} everywhere (workflow + engine)
{{/if}} → {{#endif}} in the workflow; the engine now matches {{#endif}} as primary with {{/if}} as alternate
Removed the (eq v b) operator; is_truthy.cjs now evaluates GitHub Actions script style v == "b" / v === "b" / v != "b" / v !== "b" expressions
Workflow conditions updated to experiments.prompt_style == "concise"

Copilot

Pull request overview

Implements a prompt_style A/B experiment for the daily-community-attribution workflow by adding concise/verbose prompt variants and extending the prompt templating engine to support {{#else}} / {{#endif}} and simple equality expressions in {{#if ...}} conditions.

Changes:

Add experiments: prompt_style: [concise, verbose] to the workflow and wrap major prompt sections with {{#if ...}} / {{#else}} / {{#endif}}.
Extend the template renderer to recognize {{#else}} branches and {{#endif}} closers (keeping {{/if}} as an alternate).
Add isTruthy() support + tests for lhs == "rhs" / !=-style conditions and tests for {{#else}} / {{#endif}} rendering; regenerate the workflow lockfile to include experiment selection steps.

Show a summary per file

File	Description
actions/setup/js/is_truthy.test.cjs	Adds unit tests for GitHub Actions-style equality parsing in `isTruthy()`
actions/setup/js/is_truthy.cjs	Adds parsing of `==/===` and `!=/!==` expressions before the generic truthy/falsy check
actions/setup/js/interpolate_prompt.test.cjs	Adds tests for `{{#else}}`/`{{#endif}}` rendering and equality-based condition evaluation
actions/setup/js/interpolate_prompt.cjs	Updates template rendering to support `{{#else}}` and `{{#endif}}`, and substitutes `experiments.*` references inside `{{#if ...}}` conditions
.github/workflows/daily-community-attribution.md	Adds experiment frontmatter + concise/verbose conditional variants for key workflow sections
.github/workflows/daily-community-attribution.lock.yml	Regenerates compiled lockfile including experiment pick/cache/artifact wiring and env propagation

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 6/6 changed files
Comments generated: 4

+  if (neqMatch) {
+    return neqMatch[1].trim() !== neqMatch[2];
+  }
+


+    describe("GitHub Actions script style equality (v == b)", () => {
+      it("should return true when value matches literal with ==", () => {
+        expect(isTruthy('concise == "concise"')).toBe(true);
+        expect(isTruthy('verbose == "verbose"')).toBe(true);
+        expect(isTruthy('yes == "yes"')).toBe(true);


+    //
+    // Additionally, {{#if (eq experiments.name "value")}} conditions use the dot-notation
+    // form directly in the condition expression. We substitute experiments.NAME → actual
+    // value inside {{#if ...}} condition tags so that isTruthy can evaluate (eq ...) helpers.


+- One `issue_read` per Tier 3 candidate
+- Stop after safe-output call
+- PR body under 400 words
+- No external URLs


github-actions · 2026-05-01T21:24:32Z

🧪 Test Quality Sentinel Report

Test Quality Score: 89/100

✅ Excellent test quality

Metric	Value
New/modified tests analyzed	11
✅ Design tests (behavioral contracts)	11 (100%)
⚠️ Implementation tests (low value)	0 (0%)
Tests with error/edge cases	7 (64%)
Duplicate test clusters	0
Test inflation detected	No
🚨 Coding-guideline violations	None

Test Classification Details

View All Test Classifications (11 tests)

Test	File	Classification	Issues Detected
`should keep true branch of {{#else}} block when condition is truthy`	`interpolate_prompt.test.cjs:92`	✅ Design	Covers both positive + negative assertions
`should keep false branch of {{#else}} block when condition is falsy`	`interpolate_prompt.test.cjs:98`	✅ Design	Covers both branches
`should handle {{#else}} with GitHub Actions style equality condition matching`	`interpolate_prompt.test.cjs:104`	✅ Design	Tests truthy and falsy equality expressions
`should support {{/if}} as alternate closing tag`	`interpolate_prompt.test.cjs:113`	✅ Design	Happy path only — no negative case
`should return true when value matches literal with ==`	`is_truthy.test.cjs:66`	✅ Design	Tests 3 matching pairs
`should return false when value does not match literal with ==`	`is_truthy.test.cjs:72`	✅ Design	Tests 3 non-matching pairs
`should return true when value matches literal with ===`	`is_truthy.test.cjs:79`	✅ Design	Strict equality variant
`should return false when value does not match literal with ===`	`is_truthy.test.cjs:83`	✅ Design	Negative strict equality
`should return true when value does not match literal with !=`	`is_truthy.test.cjs:87`	✅ Design	Inequality operator
`should return false when value matches literal with !=`	`is_truthy.test.cjs:91`	✅ Design	Negative inequality
`should handle whitespace around operators`	`is_truthy.test.cjs:95`	✅ Design	Edge case: whitespace tolerance

Flagged Tests — Requires Review

No tests flagged. All new tests demonstrate solid behavioral coverage.

Minor observation (not a failure): should support {{/if}} as alternate closing tag has a single toContain assertion with no negative check (e.g., confirming that {{/if}} itself is not present in the output). This is a low-severity gap — the test is still a valid design test.

Language Support

Tests analyzed:

🐹 Go (*_test.go): 0 tests
🟨 JavaScript (*.test.cjs, *.test.js): 11 tests (vitest)

Test Inflation Check

Test File	Lines Added (Test)	Lines Added (Prod)	Ratio
`interpolate_prompt.test.cjs`	25	34	0.74 — ✅
`is_truthy.test.cjs`	34	22	1.55 — ✅

Both ratios are well below the 2:1 threshold.

Verdict

✅ Check passed. 0% of new tests are implementation tests (threshold: 30%). All 11 new tests verify behavioral contracts: what isTruthy() returns for new equality-expression inputs, and what renderMarkdownTemplate() produces for {{#else}} / {{/if}} syntax. Both positive and negative branches are covered throughout.

📖 Understanding Test Classifications

Design Tests (High Value) verify what the system does:

Assert on observable outputs, return values, or state changes
Cover error paths and boundary conditions
Would catch a behavioral regression if deleted
Remain valid even after internal refactoring

Implementation Tests (Low Value) verify how the system does it:

Assert on internal function calls (mocking internals)
Only test the happy path with typical inputs
Break during legitimate refactoring even when behavior is correct
Give false assurance: they pass even when the system is wrong

Goal: Shift toward tests that describe the system's behavioral contract — the promises it makes to its users and collaborators.

References:

§25233603244

🧪 Test quality analysis by Test Quality Sentinel · ● 675.4K · ◷

github-actions

✅ Test Quality Sentinel: 89/100. Test quality is excellent — 0% of new tests are implementation tests (threshold: 30%). All 11 new tests verify behavioral contracts for the new {\{#else}}/{{/if}} template syntax and equality-expression support in isTruthy().

pelikhan · 2026-05-01T21:33:55Z

@copilot review all comments

…ording - is_truthy.cjs: fix empty-LHS edge case — change (.+?) to (.*?) so a condition like ' == "concise"' (unset experiment var) correctly returns false instead of falling through to the truthy string check - is_truthy.test.cjs: rename describe block from 'v == b' to 'v == "b"' to accurately reflect that the RHS must be quoted; add empty-LHS test cases - interpolate_prompt.cjs: update Step 2.5 comment to remove stale (eq ...) references and describe the actual lhs == "rhs" syntax used after substitution - daily-community-attribution.md: reword "No external URLs" to "Do not access external URLs; use only GitHub MCP issue_read for GitHub data" for clarity Agent-Logs-Url: https://github.com/github/gh-aw/sessions/704460b1-ad87-4898-846d-0a5207729619 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-05-01T21:39:22Z

@copilot review all comments

Addressed in commit 65a6739:

is_truthy.cjs: Fixed empty-LHS edge case — changed (.+?) to (.*?) so a condition like == "concise" (from an unset experiment variable) correctly returns false instead of falling through to the generic truthy string check.
is_truthy.test.cjs: Renamed describe block from v == b to v == "b" to accurately reflect that the RHS must be quoted; added two empty-LHS test cases.
interpolate_prompt.cjs: Updated Step 2.5 comment block to remove stale (eq ...) references; now describes the actual lhs == "rhs" syntax used after experiment substitution.
daily-community-attribution.md:457: Reworded No external URLs → Do not access external URLs; use only GitHub MCP \issue_read` for GitHub data`.

Initial plan

6b84564

Copilot AI assigned Copilot and pelikhan May 1, 2026

Copilot AI linked an issue May 1, 2026 that may be closed by this pull request

[ab-advisor] Experiment campaign for daily-community-attribution: A/B test prompt_style #29602

Closed

8 tasks

Copilot started work on behalf of pelikhan May 1, 2026 20:30 View session

Copilot AI and others added 2 commits May 1, 2026 20:42

Copilot AI changed the title ~~[WIP] Add experiment campaign for daily community attribution A/B test~~ [ab-advisor] A/B experiment: prompt_style for daily-community-attribution May 1, 2026

Copilot AI requested a review from pelikhan May 1, 2026 20:45

Copilot finished work on behalf of pelikhan May 1, 2026 20:45

pelikhan reviewed May 1, 2026

View reviewed changes

Copilot started work on behalf of pelikhan May 1, 2026 20:47 View session

Copilot finished work on behalf of pelikhan May 1, 2026 21:03

Copilot AI requested a review from pelikhan May 1, 2026 21:03

Copilot started work on behalf of pelikhan May 1, 2026 21:08 View session

github-actions Bot mentioned this pull request May 1, 2026

[Contribution Check Report] Contribution Check — 2026-05-01 #29413

Closed

Copilot finished work on behalf of pelikhan May 1, 2026 21:16

pelikhan marked this pull request as ready for review May 1, 2026 21:18

Copilot AI review requested due to automatic review settings May 1, 2026 21:18

Copilot started reviewing on behalf of pelikhan May 1, 2026 21:19 View session

Copilot AI reviewed May 1, 2026

View reviewed changes

github-actions Bot mentioned this pull request May 1, 2026

[aw] No-Op Runs #29134

Open

github-actions Bot approved these changes May 1, 2026

View reviewed changes

Copilot started work on behalf of pelikhan May 1, 2026 21:34 View session

Copilot finished work on behalf of pelikhan May 1, 2026 21:40

pelikhan merged commit ead376f into main May 1, 2026

pelikhan deleted the copilot/ab-advisor-experiment-campaign-prompt-style branch May 1, 2026 21:41

Copilot AI mentioned this pull request May 1, 2026

docs: document {{#if}} / {{#else}} / {{#endif}} template syntax in instructions files #29637

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ab-advisor] A/B experiment: prompt_style for daily-community-attribution#29620

[ab-advisor] A/B experiment: prompt_style for daily-community-attribution#29620
pelikhan merged 6 commits intomainfrom
copilot/ab-advisor-experiment-campaign-prompt-style

Copilot AI commented May 1, 2026 •

edited

Loading

Uh oh!

pelikhan May 1, 2026

Uh oh!

Copilot AI May 1, 2026

Uh oh!

pelikhan May 1, 2026

Uh oh!

Copilot AI May 1, 2026

Uh oh!

pelikhan commented May 1, 2026

Uh oh!

Copilot AI commented May 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented May 1, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

pelikhan commented May 1, 2026

Uh oh!

Copilot AI commented May 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Template Engine Additions (actions/setup/js)

Uh oh!

pelikhan May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 1, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 1, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan commented May 1, 2026

Uh oh!

Copilot AI commented May 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

github-actions Bot commented May 1, 2026

🧪 Test Quality Sentinel Report

Test Quality Score: 89/100

Test Classification Details

Flagged Tests — Requires Review

Language Support

Test Inflation Check

Verdict

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

pelikhan commented May 1, 2026

Uh oh!

Copilot AI commented May 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented May 1, 2026 •

edited

Loading

Template Engine Additions (`actions/setup/js`)