Skip to content

Runner Doctor: add D11 gVisor+Copilot CLI Node v22 startup-crash failure mode across catalog and agent surfaces#6574

Open
lpcox with Copilot wants to merge 5 commits into
mainfrom
copilot/update-runner-doctor
Open

Runner Doctor: add D11 gVisor+Copilot CLI Node v22 startup-crash failure mode across catalog and agent surfaces#6574
lpcox with Copilot wants to merge 5 commits into
mainfrom
copilot/update-runner-doctor

Conversation

Copilot AI commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Runner Doctor was missing a newly observed self-hosted failure mode: Copilot CLI startup crashes under gVisor (SIGABRT/SIGSEGV) tied to Node.js v22, with AWF currently offering only a one-shot restart mitigation. This PR adds D11 consistently so triage, quick lookup, and unresolved tracking all reflect the latest behavior.

  • Knowledge-base update (shared catalog)

    • Added D11 to shared/self-hosted-failure-modes.md (Category D table) with:
      • symptom signature (exit 139 / SIGABRT, zero-token/zero-stdout retries, occasional bash segfault),
      • root-cause framing (gVisor ↔ Node v22/V8 assertion),
      • mitigation status from #6514 (MAX_GVISOR_AGENT_RETRIES = 1, GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000),
      • probe guidance and citations (#6513, #6514, #6558).
    • Added D11 to Error-string quick lookup.
    • Added D11 to Known unresolved items.
  • Runner Doctor workflow prompt alignment

    • Updated .github/workflows/self-hosted-runner-doctor.md:
      • added D11 to “narrowest match” symptom mapping,
      • added D11 to unresolved/mitigation narrative in the notable-fixes section.
  • Portable agent parity

    • Updated .github/agents/self-hosted-runner-doctor.md with the same D11 table row, quick-lookup entry, unresolved-item entry, and symptom-mapping guidance to keep portable and workflow-backed doctor behavior in sync.
  • Sync guardrails

    • Extended scripts/ci/self-hosted-runner-doctor-workflow.test.ts assertions to require D11 presence and key strings across shared/workflow/portable surfaces.
| `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) |

Copilot AI changed the title [WIP] Update Runner Doctor for new failure mode D11 Runner Doctor: add D11 gVisor+Copilot CLI Node v22 startup-crash failure mode across catalog and agent surfaces Jul 24, 2026
Copilot finished work on behalf of lpcox July 24, 2026 22:09
Copilot AI requested a review from lpcox July 24, 2026 22:09
@lpcox
lpcox marked this pull request as ready for review July 24, 2026 23:12
Copilot AI review requested due to automatic review settings July 24, 2026 23:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds D11 guidance for diagnosing Copilot CLI Node.js v22 startup crashes under gVisor.

Changes:

  • Documents D11 symptoms, mitigation, and unresolved status.
  • Aligns workflow and portable-agent lookup guidance.
  • Adds synchronization test assertions.
Show a summary per file
File Description
.github/workflows/shared/self-hosted-failure-modes.md Adds D11 catalog guidance.
.github/workflows/self-hosted-runner-doctor.md Adds D11 workflow triage guidance.
.github/agents/self-hosted-runner-doctor.md Adds portable D11 guidance.
scripts/ci/self-hosted-runner-doctor-workflow.test.ts Checks D11 content synchronization.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 4/4 changed files
  • Comments generated: 4
  • Review effort level: Medium

| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor` or raw `runsc`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT `RETURN` bypass rule (installed by `awf-iptables-init`) that normally lets the agent reach the MCP gateway (`172.30.0.1:8080`) directly never fires under gVisor. MCP requests follow `HTTP_PROXY` into Squid and receive `403 ERR_ACCESS_DENIED` because the MCP gateway IP is not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` returns `false` for `gvisor` and its raw `runsc` alias (plus `sbx`); `awf-iptables-init` is skipped for these runtimes (`AWF_SKIP_IPTABLES_INIT=1`); the MCP gateway (`172.30.0.1`) and `host.docker.internal` are added to `NO_PROXY` so proxy-aware MCP clients connect directly. **Caveat:** proxy-unaware tools using raw sockets (e.g. `/dev/tcp`) still get `No route to host` under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for `403` on `172.30.0.1`; check `NO_PROXY` env inside the agent container — should include `172.30.0.1` when runtime is `gvisor` or raw `runsc` on patched AWF | github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 |
| D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM | AWF before PR github/gh-aw-firewall#6336 mounted the entire host `$HOME` (read-write) into the sbx microVM. Unlike compose mode (empty home volume + `/dev/null` overlays), sbx uses virtiofs passthrough where mounts are directory-granular and file-level overlays are not expressible. | **Fixed in AWF (PR github/gh-aw-firewall#6336)**: `sbx-manager.ts` mounts only whitelisted tool/cache + agent-state subdirs (`HOME_TOOL_SUBDIRS` + `.copilot`/`.gemini`). For whitelisted dirs that contain nested credential files (e.g. `~/.config/gh/hosts.yml`, `~/.cargo/credentials.toml`), `scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-<pid>` before `sbx create` and `restoreHomeCredentials()` restores them after teardown. Workaround (older AWF): do not use `--container-runtime sbx` with real credentials in `$HOME`. | `ls $HOME/.aws $HOME/.ssh $HOME/.docker 2>/dev/null` inside the sbx sandbox — if dirs are visible, fix is not applied; confirm AWF version includes github/gh-aw-firewall#6336 | github/gh-aw-firewall#6336 |
| D10 | On `--container-runtime sbx`, Copilot CLI installed via `install_copilot_cli.sh --rootless` (redirects install to `~/.local/bin`) is **not found** even though `~/.local` is mounted into the microVM; `copilot: command not found` or `ENOENT` at agent startup | sbx executes agent commands via `bash -lc` (login shell), whose profile initialization can reset PATH and discard `--env PATH=...`. The binary is present in the VM but unreachable by name unless PATH is fixed after login init. | **Fixed in AWF (PR github/gh-aw-firewall#6407, merged 2026-07-19):** sbx wraps the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` after login initialization (`withLocalBinOnPath()` in `sbx-manager.ts`). Upgrade to the AWF version including github/gh-aw-firewall#6407. Workaround (older AWF): invoke `$HOME/.local/bin/copilot` directly, or prefix the agent command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"; ...`. | Inside the sbx agent: `which copilot` or `ls ~/.local/bin/copilot` confirms binary presence; on patched AWF, inspect the executed command wrapper in sbx logs and verify it prepends `~/.local/bin` before invoking the agent command | github/gh-aw-firewall#6407 |
| D11 | Copilot CLI agent starts under `--container-runtime gvisor` but exits immediately with **exit code 139** (`SIGSEGV`) or `SIGABRT` (exit 1); `[copilot-harness]` log shows all retry attempts crashing within ~90 ms (`tokenCount=0`, `stdout=0B`); the outer `bash` wrapper can also segfault; no network calls are made. Affects `sandbox.agent.runtime: gvisor` (compose-managed gVisor) at ~8% failure rate; identical workloads on `runc` and `sbx` are unaffected. | Node.js v22 (bundled in the Copilot CLI) can trigger a V8 native assertion (`StringBytes::Encode ... Assertion failed: (written) == (u16size)`) during ESM module translation under gVisor's userspace netstack; gVisor's `mmap`/`madvise` emulation can return unexpected buffer contents to V8's UTF-8 decode path. The `SIGSEGV`/exit 139 variant can also take down the outer `bash` wrapper so the Copilot harness cannot retry. Root cause (gVisor ↔ Node.js v22 incompatibility) is **unresolved** — tracked in github/gh-aw-firewall#6558. | **Mitigated in AWF (PR github/gh-aw-firewall#6514, merged 2026-07-23):** `runAgentCommand()` detects gVisor runtime and automatically retries the agent container **once** (via `docker start awf-agent`) when the exit code is `134` (SIGABRT) or `139` (SIGSEGV) **within the first 30 s** (`GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000`); `MAX_GVISOR_AGENT_RETRIES = 1`. Log reattachment uses `docker logs --since <restart-time> -f awf-agent` to avoid replaying the crashed attempt. Non-gVisor runtimes are unaffected. Root cause remains open; upgrade AWF to include github/gh-aw-firewall#6514 to get the retry mitigation. | Agent logs show `signal=SIGABRT duration=0s stdout=0B` on attempt 1; with github/gh-aw-firewall#6514, a second `docker start awf-agent` log block appears and usually succeeds. Without the fix, all harness retries fail identically. Confirm `containerRuntime: gvisor` in the resolved `docker-compose.redacted.yml`. | github/gh-aw-firewall#6513, github/gh-aw-firewall#6514, github/gh-aw-firewall#6558 |
| D8 | MCP tool calls (`safeoutputs`, `github`) return `403 ERR_ACCESS_DENIED` under `--container-runtime gvisor` or raw `runsc`; agent completes but never writes safe outputs; smoke tests fail at "Validate safe outputs were invoked"; direct `/dev/tcp` connections fail with `No route to host` | gVisor's userspace netstack is isolated from the host network namespace. The iptables DNAT `RETURN` bypass rule (installed by `awf-iptables-init`) that normally lets the agent reach the MCP gateway (`172.30.0.1:8080`) directly never fires under gVisor. MCP requests follow `HTTP_PROXY` into Squid and receive `403 ERR_ACCESS_DENIED` because the MCP gateway IP is not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` returns `false` for `gvisor` and its raw `runsc` alias (plus `sbx`); `awf-iptables-init` is skipped for these runtimes (`AWF_SKIP_IPTABLES_INIT=1`); the MCP gateway (`172.30.0.1`) and `host.docker.internal` are added to `NO_PROXY` so proxy-aware MCP clients connect directly. **Caveat:** proxy-unaware tools using raw sockets (e.g. `/dev/tcp`) still get `No route to host` under gVisor — egress requires proxy-aware clients. | Inspect Squid access log for `403` on `172.30.0.1`; check `NO_PROXY` env inside the agent container — should include `172.30.0.1` when runtime is `gvisor` or raw `runsc` on patched AWF | github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 |
| D9 | On `--container-runtime sbx`, credential files (`~/.aws/credentials`, `~/.ssh/id_rsa`, `~/.docker/config.json`, `~/.kube/config`, `~/.azure/`, `~/.gnupg/`, `~/.netrc`, `~/.config/gh/hosts.yml`, `~/.config/gcloud/`, `~/.cargo/credentials.toml`, `~/.claude/.credentials.json`, `~/.gemini/oauth_creds.json`) are visible to the agent inside the sbx microVM | AWF before PR github/gh-aw-firewall#6336 mounted the entire host `$HOME` (read-write) into the sbx microVM. Unlike compose mode (empty home volume + `/dev/null` overlays), sbx uses virtiofs passthrough where mounts are directory-granular and file-level overlays are not expressible. | **Fixed in AWF (PR github/gh-aw-firewall#6336)**: `sbx-manager.ts` mounts only whitelisted tool/cache + agent-state subdirs (`HOME_TOOL_SUBDIRS` + `.copilot`/`.gemini`). For whitelisted dirs that contain nested credential files (e.g. `~/.config/gh/hosts.yml`, `~/.cargo/credentials.toml`), `scrubHomeCredentials()` moves them aside to `.awf-sbx-cred-backup-<pid>` before `sbx create` and `restoreHomeCredentials()` restores them after teardown. Workaround (older AWF): do not use `--container-runtime sbx` with real credentials in `$HOME`. | `ls $HOME/.aws $HOME/.ssh $HOME/.docker 2>/dev/null` inside the sbx sandbox — if dirs are visible, fix is not applied; confirm AWF version includes github/gh-aw-firewall#6336 | github/gh-aw-firewall#6336 |
| D10 | On `--container-runtime sbx`, Copilot CLI installed via `install_copilot_cli.sh --rootless` (redirects install to `~/.local/bin`) is **not found** even though `~/.local` is mounted into the microVM; `copilot: command not found` or `ENOENT` at agent startup | sbx executes agent commands via `bash -lc` (login shell), whose profile initialization can reset PATH and discard `--env PATH=...`. The binary is present in the VM but unreachable by name unless PATH is fixed after login init. | **Fixed in AWF (PR github/gh-aw-firewall#6407, merged 2026-07-19):** sbx wraps the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` after login initialization (`withLocalBinOnPath()` in `sbx-manager.ts`). Upgrade to the AWF version including github/gh-aw-firewall#6407. Workaround (older AWF): invoke `$HOME/.local/bin/copilot` directly, or prefix the agent command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"; ...`. | Inside the sbx agent: `which copilot` or `ls ~/.local/bin/copilot` confirms binary presence; on patched AWF, inspect the executed command wrapper in sbx logs and verify it prepends `~/.local/bin` before invoking the agent command | github/gh-aw-firewall#6407 |
| D11 | Copilot CLI agent starts under `--container-runtime gvisor` but exits immediately with **exit code 139** (`SIGSEGV`) or `SIGABRT` (exit 1); `[copilot-harness]` log shows all retry attempts crashing within ~90 ms (`tokenCount=0`, `stdout=0B`); the outer `bash` wrapper can also segfault; no network calls are made. Affects `sandbox.agent.runtime: gvisor` (compose-managed gVisor) at ~8% failure rate; identical workloads on `runc` and `sbx` are unaffected. | Node.js v22 (bundled in the Copilot CLI) can trigger a V8 native assertion (`StringBytes::Encode ... Assertion failed: (written) == (u16size)`) during ESM module translation under gVisor's userspace netstack; gVisor's `mmap`/`madvise` emulation can return unexpected buffer contents to V8's UTF-8 decode path. The `SIGSEGV`/exit 139 variant can also take down the outer `bash` wrapper so the Copilot harness cannot retry. Root cause (gVisor ↔ Node.js v22 incompatibility) is **unresolved** — tracked in github/gh-aw-firewall#6558. | **Mitigated in AWF (PR github/gh-aw-firewall#6514, merged 2026-07-23):** `runAgentCommand()` detects gVisor runtime and automatically retries the agent container **once** (via `docker start awf-agent`) when the exit code is `134` (SIGABRT) or `139` (SIGSEGV) **within the first 30 s** (`GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000`); `MAX_GVISOR_AGENT_RETRIES = 1`. Log reattachment uses `docker logs --since <restart-time> -f awf-agent` to avoid replaying the crashed attempt. Non-gVisor runtimes are unaffected. Root cause remains open; upgrade AWF to include github/gh-aw-firewall#6514 to get the retry mitigation. | Agent logs show `signal=SIGABRT duration=0s stdout=0B` on attempt 1; with github/gh-aw-firewall#6514, a second `docker start awf-agent` log block appears and usually succeeds. Without the fix, all harness retries fail identically. Confirm `containerRuntime: gvisor` in the resolved `docker-compose.redacted.yml`. | github/gh-aw-firewall#6513, github/gh-aw-firewall#6514, github/gh-aw-firewall#6558 |

D7 / github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 — Claude Code (Bun/JSC) crashes with `SIGSEGV`/`SIGABRT` under `--container-runtime gvisor` because JSC JIT is incompatible with gVisor's W^X memory restrictions. **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` as a manual fallback.

D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause remains unresolved (`SIGABRT` `StringBytes::Encode` assertion and occasional exit 139). **Mitigated in AWF (PR github/gh-aw-firewall#6514, merged 2026-07-23):** `runAgentCommand()` does a one-shot retry (`MAX_GVISOR_AGENT_RETRIES = 1`) when gVisor exits 134/139 within `GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000`, but this does not prevent the underlying crash.
expect(source).toContain('B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix.');
expect(source).toContain('D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED`');
expect(source).toContain('D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`.');
expect(source).toContain('D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause remains unresolved (`SIGABRT` `StringBytes::Encode` assertion and occasional exit 139).');
@lpcox

lpcox commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

@copilot address review feedback

Copilot AI commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address review feedback

Addressed in 52d4e43.

Copilot finished work on behalf of lpcox July 24, 2026 23:35
@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini completed. All facets verified. 💎

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Contribution Check completed successfully!

Contribution check complete: PR #6574 updates the relevant workflow/shared docs and adds matching test coverage; description references the related issue. No contribution-guideline issues found.

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot Network Isolation@lpcox

EGRESS_RESULT allow=pass deny=pass
  • ✅ Allowed domain (api.github.com): HTTP 200
  • ✅ Blocked domain (example.com): proxy denied (403), curl blocked

Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions github-actions Bot added the smoke-copilot-network-isolation Copilot network-isolation egress smoke test label Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) ✅

All tests passed:

  • ✅ GitHub MCP connectivity
  • ✅ GitHub.com connectivity
  • ✅ File write/read
  • ✅ BYOK inference (api-proxy → api.githubcopilot.com)

Mode: Direct BYOK (COPILOT_PROVIDER_API_KEY) via api-proxy sidecar

Status: PASS

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🔥 Smoke Test Results

Test Result
GitHub MCP connectivity ✅ Connected (secrecy policy filtered PR data)
github.com HTTP ✅ 200 OK
File write/read ⚠️ Template vars not expanded (pre-step data unavailable)

Overall: PASS (core connectivity verified)

@lpcox — smoke tests complete.

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Results — Services Connectivity

Check Result
Redis PING ❌ Name resolution failed (host.docker.internal unresolvable)
PostgreSQL pg_isready ❌ No response
PostgreSQL SELECT 1 ❌ Name resolution failed

Overall: FAILhost.docker.internal does not resolve in this environment. Service containers are unreachable.

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: API Proxy OpenTelemetry Tracing

Scenario Result Notes
S1: Module Loading otel.js loaded successfully; exports: startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, plus internal helpers
S2: Test Suite 39/39 tests passed (otel.test.js)
S3: Env Var Forwarding src/services/api-proxy-env-config.test.ts confirms GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID all forwarded to api-proxy container
S4: Token Tracker Integration onUsage callback present in token-tracker-http.js (line 406) as the OTEL hook point
S5: OTEL Diagnostics ⚠️ No OTLP endpoint configured in this run; spans written to local fallback (/var/log/api-proxy/otel.jsonl) per graceful-degradation path

All scenarios pass. OTEL tracing integration is fully functional.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.13 Python 3.12.13 ✅ YES
Node.js v24.18.0 v22.23.1 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: ❌ FAILED — Node.js version mismatch (host: v24.18.0, chroot: v22.23.1).

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

✅ chore(workflows): migrate copilot workflows to copilot-requests auth
✅ chore: upgrade gh-aw to v0.83.2 pre-release and recompile workflows
✅ GitHub title check, temp file write, Playwright title check, and build
Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Gemini Smoke Test

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

cc @lpcox

  • MCP connectivity: ❌ Error: MCP error 0: resource not accessible
  • GitHub.com connectivity: ✅ HTTP 200
  • File write/read: ❌ file not found
  • BYOK inference: ✅

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Overall: FAIL

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color 1/1 passed ✅ PASS
Go env 1/1 passed ✅ PASS
Go uuid 1/1 passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx All passed ✅ PASS
Node.js execa All passed ✅ PASS
Node.js p-limit All passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for #6574 · sonnet46 · 41 AIC · ⊞ 8.3K ·
Add label ready-for-aw to run again

The 'writes token-usage.jsonl incrementally before shutdown' test asserted
file contents after a single fixed 20ms timeout. Because writeTokenUsage()
persists via an async createWriteStream (file open + flush callback), the
record may not be flushed within 20ms on a loaded CI runner, causing the
assertion to throw inside the setTimeout callback so done() is never called
and the test times out at 5s. Poll for the record with a deadline instead,
keeping the common case fast (~1ms) while tolerating CI slowness.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4e314879-1352-440b-9c24-5dd44a918be5
@github-actions

Copy link
Copy Markdown
Contributor

@lpcox
MCP connectivity: ✅
GitHub.com connectivity: ✅
File I/O: ✅
Direct BYOK: ✅ Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)
Overall: PASS

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🔥 Smoke Test: Docker Sbx — PASS

Test Result
GitHub MCP Connectivity ✅ Connected (PRs filtered by secrecy policy — expected)
GitHub.com HTTP ✅ Reachable
File Write/Read smoke-test-docker-sbx-30134597411.txt verified

Overall: PASS@lpcox

📰 BREAKING: Report filed by Smoke Docker Sbx
Add label ready-for-aw to run again

@lpcox

lpcox commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

@copilot merge main

Copilot AI commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

@copilot merge main

Merged main in 81b553a.

Copilot finished work on behalf of lpcox July 25, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment