Skip to content

[spdd] Add spec implementation tracking, drift SLA guidance, and harness traceability updates#32653

Merged
pelikhan merged 3 commits into
mainfrom
copilot/spdd-daily-spec-work-plan
May 16, 2026
Merged

[spdd] Add spec implementation tracking, drift SLA guidance, and harness traceability updates#32653
pelikhan merged 3 commits into
mainfrom
copilot/spdd-daily-spec-work-plan

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 16, 2026

This daily SPDD issue identified missing implementation visibility in safe-output outcome specs, missing SLA tracking details for config drift, and weak spec↔implementation sync guidance across AW harness and threat-detection specs. This PR tightens those gaps with explicit status mapping, traceability, and version/sync anchors.

  • Safe-output outcome spec: implementation visibility + metrics ownership

    • Added an Implementation status column (implemented / partial / not-started) for outcome types.
    • Filled coverage across the outcome table, including explicit add_reviewer implementation references in both Go and JS areas.
    • Extended Derived Metrics with concrete ownership: Go aggregation location and OTel span emission location per metric.
  • AWF config source spec: actionable drift operations

    • Updated the Section 4.3 drift-check CLI example to current gh api path style and stronger schema-key extraction.
    • Added explicit GH token precondition in snippet.
    • Added direct workflow path link for schema consistency automation.
    • Added Drift SLA tracking subsection for CR-06 with concrete labels (workflow, bug) and escalation issue shape ([Schema Drift SLA] + owner/unblock plan/revised ETA fields).
  • AW Harness spec: budget-exhaustion recovery norm + traceability

    • Expanded safeguards for budget hard-stop behavior to define:
      • preserved durable artifacts,
      • discarded in-flight state,
      • forced-termination signaling fields.
    • Added definition for a “completed turn boundary”.
    • Added a MUST/MUST NOT traceability matrix mapping normative requirements to expected aw_harness.cjs guards.
  • Compiler threat detection spec: version-sync anchor

    • Added Spec-to-Implementation Sync section with explicit spec version ↔ minimum gh-aw binary version mapping and lock-file compatibility note.
    • Renumbered and normalized section references to keep cross-links consistent after insertion.
  • Security architecture summary: forward maintenance signal

    • Added a pending item in the Spec Maintenance table to keep next-step tracking live instead of fully closed.
  • Safe output handlers JS: spec cross-reference comments

    • Added inline cross-reference comments at handler entry points to align runtime handlers with numbered outcome sections (and mark non-numbered types explicitly).
| Output type | Implementation status | Go implementation areas | JS/runtime implementation areas |
|-------------|------------------------|--------------------------|---------------------------------|
| `add_reviewer` | not-started | `pkg/workflow/add_reviewer.go`, `pkg/workflow/safe_outputs_config.go` | `actions/setup/js/add_reviewer.cjs`, `actions/setup/js/safe_outputs_handlers.cjs` |

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh gh auth status /opt/hostedtoolcGO111MODULE -o /tmp/go-build276GOINSECURE -trimpath 64/bin/go -p golang.org/x/tooenv -lang=go1.25 go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI linked an issue May 16, 2026 that may be closed by this pull request
11 tasks
Copilot AI and others added 2 commits May 16, 2026 16:27
Copilot AI changed the title [WIP] Review daily specification files for 2026-05-16 [spdd] Add spec implementation tracking, drift SLA guidance, and harness traceability updates May 16, 2026
Copilot AI requested a review from gh-aw-bot May 16, 2026 16:31
@pelikhan pelikhan marked this pull request as ready for review May 16, 2026 16:32
Copilot AI review requested due to automatic review settings May 16, 2026 16:32
@pelikhan pelikhan merged commit 8c98ea6 into main May 16, 2026
@pelikhan pelikhan deleted the copilot/spdd-daily-spec-work-plan branch May 16, 2026 16:32
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates several SPDD specs (and adds a few inline JS cross-references) to improve spec↔implementation traceability, make config-drift handling more actionable, and anchor version/sync expectations across the harness and threat-detection specs.

Changes:

  • Added implementation status tracking and ownership/traceability details for safe-output outcome evaluation and derived metrics.
  • Strengthened AWF config drift operational guidance (CLI example, automation link, and explicit SLA escalation conventions).
  • Inserted a spec↔binary version sync section into the compiler threat-detection spec and renumbered cross-references accordingly.
Show a summary per file
File Description
specs/security-architecture-spec-summary.md Adds a pending maintenance tracker item for spec-to-lock sync guidance.
specs/safe-output-outcome-evaluation.md Adds implementation-status mapping table and expands derived-metrics ownership columns.
specs/compiler-threat-detection-spec.md Adds spec-to-implementation sync table and renumbers sections/cross-links.
specs/awf-config-sources-spec.md Updates drift-check CLI example, adds workflow reference link, and documents drift SLA escalation format.
specs/aw-harness.md Clarifies budget hard-stop semantics and adds a spec↔source traceability matrix.
actions/setup/js/safe_outputs_handlers.cjs Adds inline spec cross-reference comments at key safe-output handler entry points.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

specs/compiler-threat-detection-spec.md:288

  • This bullet references "Section 5.3.1" for the deprecation policy, but there is no §5.3.1 anchor (the doc currently contains a mismatched "#### 4.3.1 Deprecation Policy"). Update the subsection numbering/anchor (or change this reference) so the link resolves correctly.
- Each active CTR rule MUST have at least one test ID in Section 8.1 that covers the primary detection trigger.
- Tests MUST be deterministic: given the same malicious or unsafe input, the compiler MUST always emit the same diagnostic.
- Tests MUST assert the stable diagnostic ID (e.g., `CTR-006`) appears in the compiler error output so that CI can mechanically verify rule coverage.
- When a new rule is added to Section 5.1, at least one new test ID MUST be added to Section 8.1 in the same change set.
- When a rule is deprecated per Section 5.3.1, its test IDs MUST be marked `[DEPRECATED]` and removed from the required compliance gate.
  • Files reviewed: 6/6 changed files
  • Comments generated: 5

Comment on lines +133 to 139
# Fetch both schema files from gh-aw-firewall
gh api /repos/github/gh-aw-firewall/contents/docs/awf-config.schema.json \
--jq '.content' | base64 -d > /tmp/published-schema.json

gh api repos/github/gh-aw-firewall/contents/src/awf-config-schema.json \
gh api /repos/github/gh-aw-firewall/contents/src/awf-config-schema.json \
--jq '.content' | base64 -d > /tmp/runtime-schema.json

Comment on lines +140 to +152
# Extract nested schema property paths
jq -r '
def walk_props(prefix):
(.properties // {} | to_entries[]) as $p
| ($p.key) as $k
| ((if prefix == "" then $k else prefix + "." + $k end)),
($p.value | walk_props(if prefix == "" then $k else prefix + "." + $k end));
walk_props("")
' /tmp/published-schema.json | sort -u > /tmp/schema-keys.txt

# Compare against gh-aw source references
grep -rh '"apiProxy\|"network\|"model\|"auth' pkg/workflow/ | sort -u > /tmp/ghaw-refs.txt
# Compare against awf-config references in gh-aw implementation
rg --no-heading --no-filename --only-matching 'apiProxy\.[A-Za-z0-9_.]+' pkg/workflow actions/setup \
| sort -u > /tmp/ghaw-refs.txt
Comment on lines 152 to +166
@@ -150,16 +163,16 @@ When a compiler feature that a `CTR-*` rule depends on is removed, the rule MUST
- The rule's status MUST be updated to `Deprecated` in this specification in the same change set as the implementation removal.
- The rule catalog entry MUST be retained (not deleted) with a deprecation notice indicating the version in which the rule was retired and the reason.
- All test IDs mapped to the deprecated rule in Section 7 MUST be marked as `[DEPRECATED]` and MUST NOT be required for conformance after the deprecation version.
- The implementation mapping in Section 6.1 for the deprecated rule MUST be cleared; the row MUST remain in the table annotated with `[Deprecated in vX.Y.Z]`.
- The implementation mapping in Section 7.1 for the deprecated rule MUST be cleared; the row MUST remain in the table annotated with `[Deprecated in vX.Y.Z]`.
| `update_project` | not-started | `pkg/workflow/update_project.go`, `pkg/cli/outcome_eval.go` (`evalGenericSticky` fallback) | `actions/setup/js/update_project.cjs`, `actions/setup/js/evaluate_outcomes.cjs` (generic fallback) |
| `update_release` | not-started | `pkg/workflow/safe_outputs_config.go`, `pkg/cli/outcome_eval.go` (`evalGenericSticky` fallback) | `actions/setup/js/update_release.cjs`, `actions/setup/js/evaluate_outcomes.cjs` (generic fallback) |
| `noop` | implemented | `pkg/cli/outcome_eval.go` (explicit skip in `EvaluateOutcomes`) | `actions/setup/js/evaluate_outcomes.cjs` (`NOOP_TYPES`) |
| `missing_tool` | implemented | `pkg/cli/outcome_eval.go` (explicit skip in `EvaluateOutcomes`) | `actions/setup/js/missing_tool.cjs`, `actions/setup/js/evaluate_outcomes.cjs` (`NOOP_TYPES`) |
Comment on lines +619 to +624
| `acceptance_rate` | accepted / (accepted + rejected) | How often actions are kept | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) |
| `waste_rate` | rejected / total | How often actions are undone | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) |
| `ignore_rate` | ignored / total | How often actions get no response | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) |
| `zero_touch_rate` | zero_touch / accepted | How often accepted actions need no human edits | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) |
| `time_to_outcome` | median(time_to_outcome_hours) | How fast outcomes resolve | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) |
| `cost_per_accepted_outcome` | total_run_cost / accepted_count | Efficiency metric | `pkg/cli/outcome_eval.go` (`ComputeOutcomeSummary`) | `actions/setup/js/emit_outcome_spans.cjs` (`buildSummaryAttributes`) |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[spdd] Daily spec work plan - 2026-05-16

4 participants