specs: implement SPDD daily work plan 2026-07-27 (9 tasks across 5 spec files) - #48447
Merged
Merged
Conversation
10 tasks
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
|
Thanks for this contribution to the daily SPDD work plan! 🤖 This PR has a focus issue to resolve before moving forward:
Since this is marked [WIP], it looks like a work-in-progress placeholder — once the spec changes are added and the diff matches the PR body, this should be good to go.
|
- Add replace_label outcome type section (§30) to safe-output-outcome-evaluation.md with accepted/rejected/pending/N/A classification rules, OTel attributes, API failure safeguards, and references to replace-label-spec.md - Add replace_label entry to Conformance Test Table - Tighten RFC 2119 Norms section: anchor boilerplate to outcome evaluation workers, split compound MUST/SHOULD in item 3 for explicit target binding - Add §6.6 Optimizer Failure Safeguards to compiler-threat-detection-spec.md with three failure modes: API unavailability, runner timeout, rate-limit exhaustion - Add T-OT-011 backward-compatibility compliance test to otel-observability-spec.md §17 - Extract DriftRecord into new Section 3 Data Model in awf-config-sources-spec.md, renumber existing sections 3-7 to 4-8, add implementation sync notes per field, update §7.5 to cross-reference Section 3 - Add §7.4 partial-success response handling (RL-057/058/059) to replace-label-spec.md - Add cross-reference from replace-label-spec.md §9 to safe-output-outcome-evaluation.md Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Update specification files for daily SPDD work plan
specs: implement SPDD daily work plan 2026-07-27 (9 tasks across 5 spec files)
Jul 27, 2026
Contributor
🤖 PR Triage
Implements SPDD daily work plan spec tasks (9 tasks, 5 spec files). DRAFT. Spec-only changes (159 adds, 63 dels, 6 files). Low urgency, defer until undrafted and reviewed.
|
pelikhan
approved these changes
Jul 28, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Updates five specifications to address the SPDD work plan.
Changes:
- Defines
replace_labeloutcome and partial-success behavior. - Adds optimizer safeguards and OTel compatibility testing.
- Extracts the
DriftRecorddata model and renumbers related sections.
Show a summary per file
| File | Description |
|---|---|
specs/safe-output-outcome-evaluation.md |
Adds replace_label evaluation rules. |
specs/replace-label-spec.md |
Specifies partial-success response handling. |
specs/otel-observability-spec.md |
Adds backward-compatibility test T-OT-011. |
specs/compiler-threat-detection-spec.md |
Adds optimizer failure safeguards. |
specs/awf-config-sources-spec.md |
Extracts and documents DriftRecord. |
.github/workflows/smoke-copilot-auto.lock.yml |
Removes compiled model-cost metadata. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (2)
specs/compiler-threat-detection-spec.md:244
- The PR description says degraded API runs suppress both PR and issue creation, but this requirement only prohibits PRs and spec updates. The optimizer could still create an issue from incomplete data, contrary to the stated safeguard.
2. The optimizer **MUST NOT** open a pull request or update spec artifacts based on incomplete threat-coverage data obtained during a degraded API run.
specs/otel-observability-spec.md:876
- This is the only OTel change, so the linked checklist item requiring a written REASONS gap analysis for every dimension—especially exporter-unreachable safeguards—remains unfulfilled. Adding the backward-compatibility test addresses a separate checklist item.
| **T-OT-011** | §17 Backward-compatibility: v0.3.0 fields present in v0.4.0 export | All span attribute keys that were present in v0.3.0 (`gh-aw.*`, `github.*`, `gen_ai.system`, `gen_ai.usage.total_tokens`, `OTEL_EXPORTER_OTLP_ENDPOINT`, built-in setup/conclusion span names) **MUST** also be present in a v0.4.0 export at the same semantic positions | Produce a v0.4.0 compliant export from the current implementation; compare the set of attribute keys against the v0.3.0 attribute inventory (defined in Version 0.3.0 change-log entry in Section 19); assert no v0.3.0 attribute has been removed or renamed; assert no built-in span name has changed |
- Files reviewed: 6/6 changed files
- Comments generated: 9
- Review effort level: Medium
|
|
||
| ### 6.6 Optimizer Failure Safeguards | ||
|
|
||
| The daily optimizer process (Section 6) is itself subject to failure. This section specifies normative behavior for the three principal failure modes: API unavailability, runner timeout, and rate-limit or quota exhaustion. These safeguards mirror the pattern established in the AWF Config Canonical Sources Specification §7. |
Comment on lines
+235
to
+237
| ### 6.6 Optimizer Failure Safeguards | ||
|
|
||
| The daily optimizer process (Section 6) is itself subject to failure. This section specifies normative behavior for the three principal failure modes: API unavailability, runner timeout, and rate-limit or quota exhaustion. These safeguards mirror the pattern established in the AWF Config Canonical Sources Specification §7. |
|
|
||
| When the optimizer job is cancelled or exceeds its allotted execution time before completing the threat-coverage check: | ||
|
|
||
| 1. The optimizer job **MUST** emit a structured `OPTIMIZER_TIMEOUT` output entry before termination, recording the last completed step and the set of CTR rules that had not yet been evaluated at the time of cancellation. |
Comment on lines
+468
to
+471
| **RL-057**: After a successful HTTP 200 response from `setLabels`, the implementation MUST verify that the returned label array satisfies all of the following conditions: | ||
|
|
||
| 1. `label_to_add` is present in the returned label array. | ||
| 2. `label_to_remove` is absent from the returned label array (when it was present before the call and was not the same label as `label_to_add`). |
Comment on lines
+473
to
+475
| **RL-058 (Partial-success outcome)**: If either condition in RL-057 is not satisfied, the implementation MUST treat this as a `rejected` outcome and MUST log a `core.error()` entry describing which expected label condition was not met (e.g., `"replace_label: label_to_add '<name>' not found in POST-setLabels response"` or `"replace_label: label_to_remove '<name>' still present after setLabels call"`). The implementation MUST return `{ success: false, error: <message> }` in this case. | ||
|
|
||
| **RL-059 (No new error code)**: The partial-success condition described in this section MUST be treated as a `rejected` outcome using the existing `SETLABELS_FAILED` error category (Section 7.1). No new error code is introduced for partial-success; the HTTP status alone is insufficient to determine outcome correctness. |
| | **T-OT-008** | §10.2 `gh-aw.job.name` on built-in job spans | The setup span includes `gh-aw.job.name` equal to the GitHub Actions job name | Decode `/tmp/gh-aw/otel.jsonl` or captured OTLP payloads and assert the setup span attribute value exactly matches the workflow job name | | ||
| | **T-OT-009** | §10.3 `gen_ai.system` on built-in agent spans | A known engine emits a non-empty normalized provider/system value | Run a workflow with a built-in agent span, decode exported spans, and assert `gen_ai.system` is present whenever the engine mapping is known | | ||
| | **T-OT-010** | §13.3 `gh-aw.outcome.type` on outcome-evaluation spans | Outcome-evaluation spans emit the safe-output type being evaluated | Execute the outcome collector, inspect the `gh-aw.outcome.evaluate` span, and assert `gh-aw.outcome.type` matches the evaluated manifest item type | | ||
| | **T-OT-011** | §17 Backward-compatibility: v0.3.0 fields present in v0.4.0 export | All span attribute keys that were present in v0.3.0 (`gh-aw.*`, `github.*`, `gen_ai.system`, `gen_ai.usage.total_tokens`, `OTEL_EXPORTER_OTLP_ENDPOINT`, built-in setup/conclusion span names) **MUST** also be present in a v0.4.0 export at the same semantic positions | Produce a v0.4.0 compliant export from the current implementation; compare the set of attribute keys against the v0.3.0 attribute inventory (defined in Version 0.3.0 change-log entry in Section 19); assert no v0.3.0 attribute has been removed or renamed; assert no built-in span name has changed | |
Comment on lines
+253
to
+255
| ### 7.5 DriftRecord Entity Schema | ||
|
|
||
| | Property | Type | Required | Description | | ||
| |----------|------|----------|-------------| | ||
| | `property_path` | `string` | **MUST** | Dot-notation config property path (e.g., `apiProxy.anthropicAutoCache`) | | ||
| | `drift_category` | `enum` | **MUST** | One of `missing_in_ghaw`, `missing_in_schema`, or `spec_mismatch` (see Section 6.2, Step 4) | | ||
| | `suggested_action` | `string` | **MUST** | Actionable remediation text; **MUST NOT** be empty | | ||
| | `detected_at` | `string` (ISO 8601) | **MUST** | UTC timestamp of detection; filesystem-safe format **SHOULD** use `YYYY-MM-DDTHH:MM:SSZ` | | ||
| A `DriftRecord` represents a single detected schema drift item produced by the drift detection procedure (Section 7.2, Step 5). The canonical schema definition, field reference, and implementation sync notes for `DriftRecord` are in **Section 3.1**. All automation and agents that produce or consume drift reports **MUST** use the schema defined in Section 3.1 for structured drift output. |
| **Extra signals:** | ||
| - `label_to_add`: the label name that should be present after the replacement | ||
| - `label_to_remove`: the label name that should be absent after the replacement | ||
| - `zero_touch`: `accepted` and no actor-visible non-bot label changes detected after the replacement |
Comment on lines
151
to
152
| GH_AW_COMPILED_STRICT: "true" | ||
| GH_AW_INFO_MODEL_COSTS: '{"providers":{"github-copilot":{"models":{"auto":{"cost":{"input":"8.5e-07","output":"1.55e-06"}}}}}}' | ||
| GH_AW_INFO_FEATURES: '{"gh-aw-detection":false}' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
9 spec gaps identified across
safe-output-outcome-evaluation.md,compiler-threat-detection-spec.md,otel-observability-spec.md,awf-config-sources-spec.md, andreplace-label-spec.md— missing outcome type coverage, underspecified error handling, absent safeguard language, and structural data model issues.specs/safe-output-outcome-evaluation.mdreplace_label— new outcome section covering all 6 outcome categories, OTel attributes (ghaw.outcome.replace_label.*), API failure safeguards (404→rejected, 5xx→pending, rate-limit→pending), and cross-reference toreplace-label-spec.mdreplace_labelrow with pass/fail conditionsoutcome evaluation workersas the conformance target; split compound MUST/SHOULD item 3 so each keyword has its own subject clausespecs/compiler-threat-detection-spec.mdOPTIMIZER_DEGRADED, suppress PR/issue creation), runner timeout (emitOPTIMIZER_TIMEOUT, discard in-progress artifacts), rate-limit exhaustion (apply retry policy, emitOPTIMIZER_RATE_LIMITED, don't count as completed cycle)specs/otel-observability-spec.mdgh-aw.*,gen_ai.system,gen_ai.usage.total_tokens, built-in span names) remain present and unmodified in v0.4.0 exportsspecs/awf-config-sources-spec.mdDriftRecordJSON Schema out of §6.5 into a standalone entities section with implementation sync notes mapping all 4 required fields topkg/workflow/awf_config_drift_formal_test.gospecs/replace-label-spec.mdsetLabels, implementation MUST verifylabel_to_addis present andlabel_to_removeis absent in the returned array; mismatch →rejectedviaSETLABELS_FAILED; no new error code introducedspecs/safe-output-outcome-evaluation.md§30 for outcome evaluator conformance requirements