Skip to content

feat: add pi_steering_extension.cjs with automatic injection and cpi.yml integration tests#29833

Merged
pelikhan merged 8 commits intomainfrom
copilot/implement-steering-extension
May 2, 2026
Merged

feat: add pi_steering_extension.cjs with automatic injection and cpi.yml integration tests#29833
pelikhan merged 8 commits intomainfrom
copilot/implement-steering-extension

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 2, 2026

Summary

Adds actions/setup/js/pi_steering_extension.cjs, a Pi agent steering extension as described in the aw-harness specification §8.3. The extension is automatically injected into every Pi agent invocation by the gh-aw compiler — no workflow frontmatter configuration is required.

The --extension flag is appended to every pi run command in pkg/workflow/pi_engine.go:

pi run --json-log ... --extension "${RUNNER_TEMP}/gh-aw/actions/pi_steering_extension.cjs"

A new cpi.yml GitHub Actions workflow runs end-to-end integration tests for Pi extensions using Pi CLI and the Anthropic API key.

What the extension does

Implements Extension 3: Steering (Resource Pressure) from the aw-harness spec:

  • Subscribes to agent_start to record the session start time
  • Subscribes to turn_end to compute remaining time each turn
  • Injects a warning steering message when remaining time drops below GH_AW_STEERING_TIME_WARNING_MINUTES (default: 5 min)
  • Injects a critical steering message when remaining time drops below GH_AW_STEERING_TIME_CRITICAL_MINUTES (default: 2 min)
  • Each threshold fires at most once per session; when critical fires it also suppresses the warning to prevent duplicate messages

Configuration (environment variables)

Variable Default Description
GH_AW_TIMEOUT_MINUTES 30 Total allowed runtime in minutes
GH_AW_STEERING_TIME_WARNING_MINUTES 5 Minutes remaining to inject warning
GH_AW_STEERING_TIME_CRITICAL_MINUTES 2 Minutes remaining to inject critical message

Set these via engine.env in the workflow frontmatter if non-default values are needed.

cpi.yml integration test workflow

.github/workflows/cpi.yml triggers on pull_request when actions/setup/js/pi_*.cjs files change and:

  • Installs Node 24, npm dependencies, and @pi/cli
  • Runs Pi agent with ANTHROPIC_API_KEY and the steering extension loaded via --extension
  • Skips gracefully when the API key secret is not available
  • Verifies the [gh-aw/steering] initialization marker appears in Pi's output to confirm the extension loaded
  • Uploads /tmp/pi-output.log and /tmp/pi-streaming.jsonl as artifacts on every run

Files changed

  • actions/setup/js/pi_steering_extension.cjs — steering extension implementation
  • actions/setup/js/pi_steering_extension.test.cjs — 17 Vitest unit tests covering config loading, event registration, threshold logic, and once-only injection behavior
  • pkg/workflow/pi_engine.go — auto-injects --extension flag in GetExecutionSteps
  • pkg/workflow/pi_engine_test.go — updated to assert the steering extension is present in the execution step
  • .github/workflows/cpi.yml — integration test workflow for Pi extensions using Pi CLI and Anthropic key

Copilot AI and others added 2 commits May 2, 2026 21:05
…r aw-harness spec §8.3

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/a03d2041-2904-44b1-9652-b34592421793

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…ring concatenation

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/a03d2041-2904-44b1-9652-b34592421793

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented May 2, 2026

@copilot the steering extension should be added automatically to the applied arguments.

Copilot AI and others added 2 commits May 2, 2026 21:22
Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 2, 2026

@copilot the steering extension should be added automatically to the applied arguments.

Done in a672ba2. --extension "${RUNNER_TEMP}/gh-aw/actions/pi_steering_extension.cjs" is now automatically appended to every Pi execution command in GetExecutionSteps — no workflow configuration required.

Copilot AI changed the title feat: add pi_steering_extension.cjs implementing Pi agent steering per aw-harness spec §8.3 feat: add pi_steering_extension.cjs with automatic injection into Pi execution args May 2, 2026
@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented May 2, 2026

Create a chi.yml job that triggers on pi extensions files and runs integration tests using Claude anthropic key, pi agent and our extensions

@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented May 2, 2026

@copilot rename chi.yml to chi.yml

Copilot AI and others added 3 commits May 2, 2026 21:44
Agent-Logs-Url: https://github.com/github/gh-aw/sessions/7f09e465-6320-4aac-a7be-94ec441fdcae

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…thropic key

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/7f09e465-6320-4aac-a7be-94ec441fdcae

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 2, 2026

Create a chi.yml job that triggers on pi extensions files and runs integration tests using Claude anthropic key, pi agent and our extensions

Done in 551fc17. .github/workflows/chi.yml triggers on pull_request when actions/setup/js/pi_*.cjs files change, installs Pi CLI, and runs Pi with the steering extension using ANTHROPIC_API_KEY. The job skips gracefully when the key is absent and verifies the [gh-aw/steering] marker in the output to confirm the extension loaded.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv --local core.hooksPath repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv /ref/tags/v9 core.hooksPath sv (http block)
  • https://api.github.com/repos/actions/download-artifact/git/ref/tags/v8
    • Triggering command: /usr/bin/gh gh api /repos/actions/download-artifact/git/ref/tags/v8 --jq [.object.sha, .object.type] | @tsv get --local $name) { hasDiscussionsEnabled } } commit.gpgsign (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/download-artifact/git/ref/tags/v8 --jq [.object.sha, .object.type] | @tsv get --local rgo/bin/bash commit.gpgsign (http block)
  • https://api.github.com/repos/actions/github-script/git/ref/tags/v9
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv com&gt; (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv .&#34; (http block)
  • https://api.github.com/repos/actions/setup-node/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv ithub/workflows core.hooksPath bin/sh (http block)
  • https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv (http block)
  • https://api.github.com/repos/docker/metadata-action/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/docker/metadata-action/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv --local --get ache/uv/0.11.8/x86_64/bash (http block)
    • Triggering command: /usr/bin/gh gh api /repos/docker/metadata-action/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv --local --get 64/bin/git (http block)
  • https://api.github.com/repos/docker/setup-buildx-action/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/docker/setup-buildx-action/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --local --get n-dir/git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/docker/setup-buildx-action/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --local --get ache/node/24.14.1/x64/bin/bash (http block)
  • https://api.github.com/repos/github/gh-aw
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .default_branch ithub/workflows copilot/implemenowner=github k/gh-aw/gh-aw/.g-f (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI changed the title feat: add pi_steering_extension.cjs with automatic injection into Pi execution args feat: add pi_steering_extension.cjs with automatic injection and chi.yml integration tests May 2, 2026
@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented May 2, 2026

@copilot rename chi.yml to chi.yml

Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 2, 2026

@copilot rename chi.yml to chi.yml

The file is already named chi.yml. Could you clarify the intended target name? It looks like the source and destination in the rename request are both chi.yml.

@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented May 2, 2026

@copilot rename chi.yml to cpi.yml

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/faf0ae2d-3c89-453a-a2c5-1c2111acfaf1

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 2, 2026

@copilot rename chi.yml to cpi.yml

Done in 83c4f47. .github/workflows/chi.yml has been renamed to .github/workflows/cpi.yml.

Copilot AI changed the title feat: add pi_steering_extension.cjs with automatic injection and chi.yml integration tests feat: add pi_steering_extension.cjs with automatic injection and cpi.yml integration tests May 2, 2026
@pelikhan pelikhan marked this pull request as ready for review May 2, 2026 22:42
Copilot AI review requested due to automatic review settings May 2, 2026 22:42
@pelikhan pelikhan merged commit 11c094e into main May 2, 2026
23 of 27 checks passed
@pelikhan pelikhan deleted the copilot/implement-steering-extension branch May 2, 2026 22:42
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a built-in Pi steering extension and wires it into Pi agent execution, plus new unit/integration coverage around the extension behavior.

Changes:

  • Auto-injects pi_steering_extension.cjs into Pi CLI invocations generated by the workflow engine.
  • Adds a new steering extension implementation and Vitest unit tests for config loading, event registration, and one-time warning/critical injection behavior.
  • Adds a GitHub Actions workflow intended to run Pi extension integration checks, and updates the checked-in dev lock workflow to include the injected extension.
Show a summary per file
File Description
pkg/workflow/pi_engine_test.go Updates Pi engine execution-step assertions to expect the built-in extension path.
pkg/workflow/pi_engine.go Appends the built-in steering extension to generated pi run commands.
actions/setup/js/pi_steering_extension.test.cjs Adds unit tests for config parsing and steering threshold behavior.
actions/setup/js/pi_steering_extension.cjs Implements the Pi steering extension and environment-driven threshold config.
.github/workflows/dev.lock.yml Regenerates locked workflow output so Pi execution includes the extension.
.github/workflows/cpi.yml Adds a new Pi integration workflow to install Pi CLI, run the extension, and upload logs.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 6/6 changed files
  • Comments generated: 3

Comment thread .github/workflows/cpi.yml
types: [opened, synchronize, reopened, ready_for_review]
paths:
- 'actions/setup/js/pi_*.cjs'
- '.github/workflows/chi.yml'
Comment thread .github/workflows/cpi.yml
if grep -q "\[gh-aw/steering\]" /tmp/pi-output.log; then
echo "✅ Steering extension initialized (found [gh-aw/steering] in output)" >> $GITHUB_STEP_SUMMARY
else
echo "⚠️ [gh-aw/steering] marker not found in output — extension may not have been loaded" >> $GITHUB_STEP_SUMMARY
Comment on lines +35 to +37
const timeoutMinutes = parseFloat(process.env.GH_AW_TIMEOUT_MINUTES || "") || DEFAULT_TIMEOUT_MINUTES;
const timeWarningMinutes = parseFloat(process.env.GH_AW_STEERING_TIME_WARNING_MINUTES || "") || DEFAULT_TIME_WARNING_MINUTES;
const timeCriticalMinutes = parseFloat(process.env.GH_AW_STEERING_TIME_CRITICAL_MINUTES || "") || DEFAULT_TIME_CRITICAL_MINUTES;
@github-actions github-actions Bot mentioned this pull request May 2, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 2, 2026

🧪 Test Quality Sentinel Report

Test Quality Score: 63/100

⚠️ Acceptable — with suggestions

Metric Value
New/modified tests analyzed 17 JS + 1 Go assertion
✅ Design tests (behavioral contracts) 17 (94%)
⚠️ Implementation tests (low value) 1 (6%)
Tests with error/edge cases 9 (53%)
Duplicate test clusters 0
Test inflation detected ✅ YES — 326 test lines vs 95 production lines (3.4:1)
🚨 Coding-guideline violations None

Test Classification Details

View All 17 JavaScript Test Classifications
Test File Classification Notes
should return default values when no env vars are set pi_steering_extension.test.cjs ✅ Design Verifies default config values
should read timeout from GH_AW_TIMEOUT_MINUTES pi_steering_extension.test.cjs ✅ Design Verifies env var parsing
should read warning threshold from GH_AW_STEERING_TIME_WARNING_MINUTES pi_steering_extension.test.cjs ✅ Design Verifies env var parsing
should read critical threshold from GH_AW_STEERING_TIME_CRITICAL_MINUTES pi_steering_extension.test.cjs ✅ Design Verifies env var parsing
should fall back to defaults for non-numeric env var values pi_steering_extension.test.cjs ✅ Design Edge case: invalid inputs
should support fractional minute values pi_steering_extension.test.cjs ✅ Design Edge case: fractional inputs
should register agent_start and turn_end handlers pi_steering_extension.test.cjs ⚠️ Implementation Only asserts .on() called with specific strings
should log session start to stderr pi_steering_extension.test.cjs ✅ Design Verifies observable side effect
should not steer when plenty of time remains pi_steering_extension.test.cjs ✅ Design Boundary: no action when above threshold
should inject a warning message when time remaining falls below warning threshold pi_steering_extension.test.cjs ✅ Design Behavioral contract with full output assertions
should inject a critical message when time remaining falls below critical threshold pi_steering_extension.test.cjs ✅ Design Behavioral contract
should only inject the warning message once even across multiple turns pi_steering_extension.test.cjs ✅ Design Idempotency invariant
should only inject the critical message once even across multiple turns pi_steering_extension.test.cjs ✅ Design Idempotency invariant
should not steer when agent_start has not fired yet pi_steering_extension.test.cjs ✅ Design Edge case: uninitialized state
should log to stderr when injecting warning message pi_steering_extension.test.cjs ✅ Design Observable logging behavior
should log to stderr when injecting critical message pi_steering_extension.test.cjs ✅ Design Observable logging behavior
should inject warning before critical when time drops through both thresholds pi_steering_extension.test.cjs ✅ Design Sequential threshold state machine

Go test file (pi_engine_test.go, +1 line): 1 new assertion added to TestPiEngine_GetExecutionSteps_Basic — verifies that the execution step auto-loads pi_steering_extension.cjs. ✅ Design test.


Flagged Tests — Requires Review

⚠️ should register agent_start and turn_end handlers (pi_steering_extension.test.cjs)

Classification: Implementation test
Issue: Only asserts that mockPi.on was called with "agent_start" and "turn_end" — it does not verify any observable behavior triggered by those events. The other tests in the suite already exercise this indirectly (they call handlers["agent_start"] directly, which would fail if registration didn't work).
What design invariant does this test enforce? None that isn't already covered by the event handler tests.
What would break if deleted? A behavioral regression (wrong event names) would still be caught by the other tests that invoke handlers["agent_start"] and handlers["turn_end"].
Suggested improvement: Either delete this test as redundant, or expand it to assert on the outcome of handler registration — e.g., that calling the registered agent_start handler produces observable output.


Test Inflation Note

pi_steering_extension.test.cjs adds 326 lines of tests for 95 lines of production code (pi_steering_extension.cjs) — a 3.4:1 ratio (threshold: 2:1). This is the primary reason for the score penalty.

This isn't necessarily bad — comprehensive testing of a stateful timer/event module often requires verbose setup. However, the repetitive per-test setup across all turn_end handler tests (creating handlers, steerCalls, mockPi, mockCtx) could be extracted into a beforeEach helper, which would reduce test file size without losing coverage.


Language Support

Tests analyzed:

  • 🐹 Go (*_test.go): 1 assertion added to existing test (//go:build !integration ✅)
  • 🟨 JavaScript (*.test.cjs): 17 tests (vitest)

Verdict

Check passed. 5.9% of new tests are implementation tests (threshold: 30%). No coding-guideline violations detected.


📖 Understanding Test Classifications

Design Tests (High Value) verify what the system does:

  • Assert on observable outputs, return values, or state changes
  • Cover error paths and boundary conditions
  • Would catch a behavioral regression if deleted
  • Remain valid even after internal refactoring

Implementation Tests (Low Value) verify how the system does it:

  • Assert on internal function calls (mocking internals)
  • Only test the happy path with typical inputs
  • Break during legitimate refactoring even when behavior is correct
  • Give false assurance: they pass even when the system is wrong

Goal: Shift toward tests that describe the system's behavioral contract — the promises it makes to its users and collaborators.

References: §25263713763

🧪 Test quality analysis by Test Quality Sentinel · ● 499.3K ·

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Test Quality Sentinel: 63/100. Test quality is acceptable — 5.9% of new tests are implementation tests (threshold: 30%).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants