[spdd] Add spec implementation tracking, drift SLA guidance, and harness traceability updates#32653
Merged
Merged
Conversation
11 tasks
Agent-Logs-Url: https://github.com/github/gh-aw/sessions/d183932a-d0fd-481d-82bc-5a7cea88da00 Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Agent-Logs-Url: https://github.com/github/gh-aw/sessions/d183932a-d0fd-481d-82bc-5a7cea88da00 Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Review daily specification files for 2026-05-16
[spdd] Add spec implementation tracking, drift SLA guidance, and harness traceability updates
May 16, 2026
pelikhan
approved these changes
May 16, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates several SPDD specs (and adds a few inline JS cross-references) to improve spec↔implementation traceability, make config-drift handling more actionable, and anchor version/sync expectations across the harness and threat-detection specs.
Changes:
- Added implementation status tracking and ownership/traceability details for safe-output outcome evaluation and derived metrics.
- Strengthened AWF config drift operational guidance (CLI example, automation link, and explicit SLA escalation conventions).
- Inserted a spec↔binary version sync section into the compiler threat-detection spec and renumbered cross-references accordingly.
Show a summary per file
| File | Description |
|---|---|
specs/security-architecture-spec-summary.md |
Adds a pending maintenance tracker item for spec-to-lock sync guidance. |
specs/safe-output-outcome-evaluation.md |
Adds implementation-status mapping table and expands derived-metrics ownership columns. |
specs/compiler-threat-detection-spec.md |
Adds spec-to-implementation sync table and renumbers sections/cross-links. |
specs/awf-config-sources-spec.md |
Updates drift-check CLI example, adds workflow reference link, and documents drift SLA escalation format. |
specs/aw-harness.md |
Clarifies budget hard-stop semantics and adds a spec↔source traceability matrix. |
actions/setup/js/safe_outputs_handlers.cjs |
Adds inline spec cross-reference comments at key safe-output handler entry points. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (1)
specs/compiler-threat-detection-spec.md:288
- This bullet references "Section 5.3.1" for the deprecation policy, but there is no §5.3.1 anchor (the doc currently contains a mismatched "#### 4.3.1 Deprecation Policy"). Update the subsection numbering/anchor (or change this reference) so the link resolves correctly.
- Each active CTR rule MUST have at least one test ID in Section 8.1 that covers the primary detection trigger.
- Tests MUST be deterministic: given the same malicious or unsafe input, the compiler MUST always emit the same diagnostic.
- Tests MUST assert the stable diagnostic ID (e.g., `CTR-006`) appears in the compiler error output so that CI can mechanically verify rule coverage.
- When a new rule is added to Section 5.1, at least one new test ID MUST be added to Section 8.1 in the same change set.
- When a rule is deprecated per Section 5.3.1, its test IDs MUST be marked `[DEPRECATED]` and removed from the required compliance gate.
- Files reviewed: 6/6 changed files
- Comments generated: 5
Comment on lines
+133
to
139
| # Fetch both schema files from gh-aw-firewall | ||
| gh api /repos/github/gh-aw-firewall/contents/docs/awf-config.schema.json \ | ||
| --jq '.content' | base64 -d > /tmp/published-schema.json | ||
|
|
||
| gh api repos/github/gh-aw-firewall/contents/src/awf-config-schema.json \ | ||
| gh api /repos/github/gh-aw-firewall/contents/src/awf-config-schema.json \ | ||
| --jq '.content' | base64 -d > /tmp/runtime-schema.json | ||
|
|
Comment on lines
+140
to
+152
| # Extract nested schema property paths | ||
| jq -r ' | ||
| def walk_props(prefix): | ||
| (.properties // {} | to_entries[]) as $p | ||
| | ($p.key) as $k | ||
| | ((if prefix == "" then $k else prefix + "." + $k end)), | ||
| ($p.value | walk_props(if prefix == "" then $k else prefix + "." + $k end)); | ||
| walk_props("") | ||
| ' /tmp/published-schema.json | sort -u > /tmp/schema-keys.txt | ||
|
|
||
| # Compare against gh-aw source references | ||
| grep -rh '"apiProxy\|"network\|"model\|"auth' pkg/workflow/ | sort -u > /tmp/ghaw-refs.txt | ||
| # Compare against awf-config references in gh-aw implementation | ||
| rg --no-heading --no-filename --only-matching 'apiProxy\.[A-Za-z0-9_.]+' pkg/workflow actions/setup \ | ||
| | sort -u > /tmp/ghaw-refs.txt |
Comment on lines
152
to
+166
| @@ -150,16 +163,16 @@ When a compiler feature that a `CTR-*` rule depends on is removed, the rule MUST | |||
| - The rule's status MUST be updated to `Deprecated` in this specification in the same change set as the implementation removal. | |||
| - The rule catalog entry MUST be retained (not deleted) with a deprecation notice indicating the version in which the rule was retired and the reason. | |||
| - All test IDs mapped to the deprecated rule in Section 7 MUST be marked as `[DEPRECATED]` and MUST NOT be required for conformance after the deprecation version. | |||
| - The implementation mapping in Section 6.1 for the deprecated rule MUST be cleared; the row MUST remain in the table annotated with `[Deprecated in vX.Y.Z]`. | |||
| - The implementation mapping in Section 7.1 for the deprecated rule MUST be cleared; the row MUST remain in the table annotated with `[Deprecated in vX.Y.Z]`. | |||
| | `update_project` | not-started | `pkg/workflow/update_project.go`, `pkg/cli/outcome_eval.go` (`evalGenericSticky` fallback) | `actions/setup/js/update_project.cjs`, `actions/setup/js/evaluate_outcomes.cjs` (generic fallback) | | ||
| | `update_release` | not-started | `pkg/workflow/safe_outputs_config.go`, `pkg/cli/outcome_eval.go` (`evalGenericSticky` fallback) | `actions/setup/js/update_release.cjs`, `actions/setup/js/evaluate_outcomes.cjs` (generic fallback) | | ||
| | `noop` | implemented | `pkg/cli/outcome_eval.go` (explicit skip in `EvaluateOutcomes`) | `actions/setup/js/evaluate_outcomes.cjs` (`NOOP_TYPES`) | | ||
| | `missing_tool` | implemented | `pkg/cli/outcome_eval.go` (explicit skip in `EvaluateOutcomes`) | `actions/setup/js/missing_tool.cjs`, `actions/setup/js/evaluate_outcomes.cjs` (`NOOP_TYPES`) | |
Comment on lines
+619
to
+624
| | `acceptance_rate` | accepted / (accepted + rejected) | How often actions are kept | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) | | ||
| | `waste_rate` | rejected / total | How often actions are undone | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) | | ||
| | `ignore_rate` | ignored / total | How often actions get no response | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) | | ||
| | `zero_touch_rate` | zero_touch / accepted | How often accepted actions need no human edits | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) | | ||
| | `time_to_outcome` | median(time_to_outcome_hours) | How fast outcomes resolve | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) | | ||
| | `cost_per_accepted_outcome` | total_run_cost / accepted_count | Efficiency metric | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This daily SPDD issue identified missing implementation visibility in safe-output outcome specs, missing SLA tracking details for config drift, and weak spec↔implementation sync guidance across AW harness and threat-detection specs. This PR tightens those gaps with explicit status mapping, traceability, and version/sync anchors.
Safe-output outcome spec: implementation visibility + metrics ownership
implemented/partial/not-started) for outcome types.add_reviewerimplementation references in both Go and JS areas.AWF config source spec: actionable drift operations
gh apipath style and stronger schema-key extraction.workflow,bug) and escalation issue shape ([Schema Drift SLA]+ owner/unblock plan/revised ETA fields).AW Harness spec: budget-exhaustion recovery norm + traceability
aw_harness.cjsguards.Compiler threat detection spec: version-sync anchor
Security architecture summary: forward maintenance signal
Safe output handlers JS: spec cross-reference comments
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
https://api.github.com/graphql/usr/bin/gh gh auth status /opt/hostedtoolcGO111MODULE -o /tmp/go-build276GOINSECURE -trimpath 64/bin/go -p golang.org/x/tooenv -lang=go1.25 go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)If you need me to access, download, or install something from one of these locations, you can either: