Repository navigation
release validation
Date: 2026-10-09
Tracking: Issue #156, under #149.
Umbrella plan: Codex App Server Update and Remote Connection, Phase 7.
Baseline: main commit b44e856; Worker contract v21; CLI 0.159.1; regression fixtures 0.155.1.
Status: Approved on 2026-10-09; implementation PR #172 is open; release acceptance remains incomplete.
Produce reproducible evidence for the complete App Server update at a specific release-candidate commit. Reuse the completed protocol, recovery, ownership, interaction, path, and package work. The verification plan distinguishes local acceptance from external services that public CI cannot exercise.
PR #170 recorded Debug/Release builds, Core 399 passed / 5 skipped, UI 370 passed / 1 skipped, schema/cache/method checks, and VSIX/XAML/package hashes. Those are historical results for that revision. They do not establish that a later release candidate passed. Issue #155 is closed after PR #170; unfinished Experimental Instance visual/accessibility acceptance is carried into #156 and remains Local-required.
| Class | Scenarios | Completion rule |
|---|---|---|
| Local-required | Pinned schemas, Fake App Server, Core/UI tests, local transport/TLS/token-rotation simulations, path boundaries, VSIX, Experimental Instance, and two instances on one machine | Every required scenario passes. A required skip or blocked result is incomplete. |
| External | Authenticated pinned CLI traffic, live MCP OAuth expiry/reauthentication, and a real remote endpoint/certificate/token rotation | Pass, or a specifically accepted blocked limitation with risk and evidence recorded. |
Record status as passed, failed, blocked, not-run, or flaky. Any failed, flaky, or not-run blocks readiness. External blocked requires the scenario, reason, unverified risk, substitute evidence, approver, date, and approval reference. It is not an automatic waiver. Local-required blocked remains incomplete.
- Generate and structurally compare CLI 0.155.1 and 0.159.1 stable/experimental schemas; verify cache and used-method surfaces. Fake traffic and the pinned CLI live-traffic scenario are separate evidence.
- Cover unknown/malformed/missing/null protocol values, completion-before-response, history notifications, duplicate items, resolved-response races, and stale generations.
- Exercise worker/transport/authentication failure, overload, reconnect exhaustion, zero mutation replay, local loopback policy, simulated TLS, and health/RPC disagreement.
- Verify Windows/POSIX path mapping, traversal/sibling-prefix/symlink/junction boundaries, mapped actions, attachment paging/recovery, and owner/principal separation.
- Run two Visual Studio Experimental Instances on the same machine and verify conversation, drafts, catalog, authentication, pending requests, and approvals do not cross instances.
- Cover question/permission/MCP flows, Gateway OAuth, cancellation and resolved races, secret refusal, and failed-tool non-replay.
- Native verification remains unsupported: prove capability is undeclared, requests are refused before projection, challenge/proof never leaks, delayed resolution is ignored, and principal changes discard pending state. Do not require enroll/verify/cancel/delete success flows.
Map every criterion to an existing test or a named remaining test before implementation. Add only missing cases.
Use a PowerShell 7 orchestrator on Windows to run the pinned CLI hash check; schema generation for both versions and surfaces; comparison, cache, and method checks; Debug/Release builds; Core/UI suites; and VSIX inspection. Limit CODEX_PATH to the steps that require it.
Compare packaged manifest, Worker payload, contract version, DLLs, and embedded XAML with the matching build outputs. Record commit, environment/tool versions, CLI hash, scenario/class/status, command, exit code, counts, skip reasons, retry outcome, and artifact hashes in a result manifest.
Both CI and Release workflows preserve the first attempt and retry only failed tests once. A retry pass is flaky and fails the gate. If an exact data-driven row cannot be isolated or the test host exits abnormally, record the error instead of retrying the whole suite. With if: always(), upload sanitized TRX, diagnostics, schema reports, hashes, and the manifest. Never publish raw authenticated traffic.
Probe symlink/junction creation in a dedicated temporary directory under the actual test identity. If a required link cannot be created, fail setup and record why. Record every skipped test and its actual reason; do not infer the six PR #170 skips without their TRX.
Run authenticated pinned CLI traffic, live MCP OAuth expiry/reauthentication, and real remote TLS/certificate/token-rotation scenarios manually outside public CI. Use a dedicated workspace and runtime-supplied endpoints. Save a redacted summary and manifest only; do not retain credentials, tokens, or raw authenticated traffic.
Use the SDK-owned F5/Experimental Instance workflow and duplicate-identity guard. Inspect Light, Dark, High Contrast, narrow layouts, connection/recovery/history/draft, question/permission/MCP, artifact, shell-confirmation, and Windows-setup states. Verify keyboard operation, focus order, accessible names, Narrator/Accessibility Insights announcements, long-history operation, and same-machine instance isolation.
Link screenshots to scenario ID, environment, expected result, and outcome. Keyboard and screen-reader checks also require recorded steps and observations. Off-screen WPF renders are supplementary only. If the Experimental Instance is unavailable, visual acceptance stays incomplete and Local-required.
Full acceptance runs on Windows with PowerShell 7. Core/Worker target net8.0, but platform-specific cases may be inconclusive elsewhere. UI/WPF tests, pinned Windows CLI schema generation, VSIX packaging, and Experimental Instance checks require Windows. Bash examples may cover portable Core/schema-report tasks or syntax in a Windows-capable Bash environment; they do not imply Linux support for the Windows CLI or UI suite.
The result manifest and approved limitations feed the final record in doc/implementation.md and doc/task.md. Readiness requires every Local-required pass, no failed/flaky/not-run, and each External scenario passed or explicitly accepted as blocked. Closed #150–#155 issues do not replace unfinished acceptance inherited here.
Candidate commit: 712d6e19b10683fb2dcc639369e040c3600c80ab. Implementation and CI/release integration are under review in PR #172. The implementation adds an evidence-producing Windows verifier and retry/skip policy; the result manifest is tied to this candidate commit.
PR CI run 116 passed on merge commit 76599cf9d03545a03d6666875ef441edcf7114f9, which combines this PR head with baseline 44e856. The hosted Windows runner had symlink capability and reported all Core/UI suites passed. Its sanitized evidence artifact is retained with the workflow run. The exact-candidate local run below remains separately recorded as blocked for the current identity.
Passed: CLI 0.155.1 and 0.159.1 hash checks; four stable/experimental schema generations; schema cache and method-surface checks; synthetic retry-policy cases; Debug and Release builds with zero warnings/errors; VSIX payload inspection including Worker contract v21 and embedded XAML.
Blocked Local-required: Debug and Release Core each reported 403 passed, 0 failed, 5 skipped of 408. UI each reported 375 passed, 0 failed, 1 skipped of 376. All six skips require symlink creation. The current Windows identity cannot create file or directory symlinks; junction creation succeeded. The suites and overall matrix remain blocked, as required.
Not run: Experimental Instance visual/accessibility and same-machine two-instance acceptance; authenticated pinned CLI traffic; live MCP OAuth expiry/reauthentication; real remote TLS/certificate/token rotation. No External limitation or waiver has been accepted. Release readiness is not declared.
See the candidate status on Issue #156 and the English evidence map with its Japanese counterpart.
- Issue #156
- PR #170 — historical implementation and automated evidence
- Codex App Server Update and Remote Connection