docs(spdd): strengthen safeguards, OTel norms, and compliance tests across 5 spec files#33096
Merged
Merged
Conversation
10 tasks
Agent-Logs-Url: https://github.com/github/gh-aw/sessions/fdf8aa77-db27-4076-ad17-4466c19979ec Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
…ce tests Agent-Logs-Url: https://github.com/github/gh-aw/sessions/fdf8aa77-db27-4076-ad17-4466c19979ec Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Update daily spec work plan for 2026-05-18
docs(spdd): strengthen safeguards, OTel norms, and compliance tests across 5 spec files
May 18, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Updates multiple SPDD specification documents to tighten security safeguards, clarify observability/OTel norms, and expand compliance testing guidance, including adding a new forecast compliance fixture directory.
Changes:
- Added/strengthened normative spec requirements: MCP Scripts stdout secret redaction, Forecast minimum sample-size + rate-limit safeguards, and Effective Tokens OTel attribute keys + new aggregation test.
- Added
specs/forecast-compliance-fixtures/with a minimalRunSummaryJSON fixture and instructions for running forecast compliance tests. - Regenerated/normalized several generated artifacts and config files (workflow lock files, editor autocomplete JSON, JS handler tests, and skill SSL JSON formatting).
Show a summary per file
| File | Description |
|---|---|
| specs/forecast-compliance-fixtures/run_summary_minimal.json | New minimal RunSummary fixture for forecast compliance tests |
| specs/forecast-compliance-fixtures/README.md | Documents fixture usage + how to run forecast compliance tests |
| pkg/workflow/safe_outputs_app_config.go | Comment update near GitHub expression non-empty check helper |
| docs/src/content/docs/reference/mcp-scripts-specification.md | Adds normative stdout sanitization requirements + new negative compliance tests |
| docs/src/content/docs/reference/forecast-specification.md | Adds sample-size warning norms and API rate-limit backoff/partial-result requirements; references fixtures |
| docs/src/content/docs/reference/effective-tokens-specification.md | Adds normative OTel attribute keys + new zero-ET leaf aggregation test definition |
| docs/src/content/docs/reference/fuzzy-schedule-specification.md | Specifies FNV-1a 32-bit as normative hash for cross-platform consistency tests |
| docs/public/editor/autocomplete-data.json | Formatting-only normalization of enum arrays |
| actions/setup/js/safe_outputs_handlers.test.cjs | Formatting-only changes to assertions |
| actions/setup/js/safe_outputs_handlers.cjs | Minor formatting change around review event normalization |
| actions/setup/js/safe_output_handler_manager.cjs | Formatting-only change in conditional |
| actions/setup/js/pr_review_buffer.cjs | Formatting-only change in error message construction |
| .github/workflows/pr-description-caveman.lock.yml | Regenerated lock workflow (setup changes, OTel plumbing, container refs, etc.) |
| .github/workflows/aw-portfolio-yield.lock.yml | Updates OTel service naming/headers and regenerated lock workflow details |
| .github/skills/ssl/ssl.json | JSON formatting normalization |
| .github/skills/reporting/ssl.json | JSON formatting normalization |
| .github/skills/jqschema/ssl.json | JSON formatting normalization |
| .github/skills/error-messages/ssl.json | JSON formatting normalization |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 18/19 changed files
- Comments generated: 4
Comment on lines
+130
to
+131
| // expressions (`${{ secrets.KEY }}` -> `secrets.KEY != ”`) or literals | ||
| // (`plain-value` -> `'plain-value' != ”`). |
| |---|---|---| | ||
| | `run_summary_zero_et.json` | Run with missing/zero ET (artifact not downloaded) | T-FC-022 | | ||
| | `run_summary_failed.json` | Run with `conclusion: "failure"` for Bernoulli sampling | T-FC-035 | | ||
| | `run_summary_high_et.json` | Run with very high ET (≥ 1,000,000) for overflow checks | T-ET-006 | |
Comment on lines
+1
to
5
| # gh-aw-metadata: {"schema_version":"v3","frontmatter_hash":"d323adc7f9e87e820faddb72dc94b0db9f74299868e4ca982ddcecc4a4edc777","strict":true,"agent_id":"copilot"} | ||
| # gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"de0fac2e4500dabe0009e67214ff5f5447ce83dd","version":"v6.0.2"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e","version":"v6.4.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.25.46"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.25.46"},{"image":"ghcr.io/github/gh-aw-firewall/cli-proxy:0.25.46"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.25.46"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.3.9","digest":"sha256:64828b42a4482f58fab16509d7f8f495a6d97c972a98a68aff20543531ac0388","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.3.9@sha256:64828b42a4482f58fab16509d7f8f495a6d97c972a98a68aff20543531ac0388"},{"image":"ghcr.io/github/github-mcp-server:v1.0.4"},{"image":"node:lts-alpine","digest":"sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f","pinned_image":"node:lts-alpine@sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f"}]} | ||
| # ___ _ _ | ||
| # / _ \ | | (_) | ||
| # | |_| | __ _ ___ _ __ | |_ _ ___ |
Comment on lines
41
to
+45
| # Container images used: | ||
| # - ghcr.io/github/gh-aw-firewall/agent:0.25.41@sha256:cb2b565d070116d4b67e355775340528b5a2c3cb18b2c9049638bcc2df681770 | ||
| # - ghcr.io/github/gh-aw-firewall/api-proxy:0.25.41@sha256:fadd0de387209f69a9a7a1b8722bb5e7fdfb80ba9749a5c60f0e4cd7582a74d0 | ||
| # - ghcr.io/github/gh-aw-firewall/cli-proxy:0.25.41@sha256:62171f2fa508667b8b0a9e096f826983f312e3da0ce894f80c0f83a875af60fe | ||
| # - ghcr.io/github/gh-aw-firewall/squid:0.25.41@sha256:1260445d25968dbf3ae70143964177a0e5914cf2ce07a6117f7d3caec6c3e3c4 | ||
| # - ghcr.io/github/gh-aw-mcpg:v0.3.6@sha256:2bb8eef86006a4c5963c55616a9c51c32f27bfdecb023b8aa6f91f6718d9171c | ||
| # - ghcr.io/github/github-mcp-server:v1.0.3@sha256:2ac27ef03461ef2b877031b838a7d1fd7f12b12d4ace7796d8cad91446d55959 | ||
| # - ghcr.io/github/gh-aw-firewall/agent:0.25.46 | ||
| # - ghcr.io/github/gh-aw-firewall/api-proxy:0.25.46 | ||
| # - ghcr.io/github/gh-aw-firewall/cli-proxy:0.25.46 | ||
| # - ghcr.io/github/gh-aw-firewall/squid:0.25.46 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SPDD review identified P0 security gaps (missing secret-redaction norms in MCP Scripts, no rate-limit safeguards in Forecast), P1 norm gaps (weak OTel attribute definitions, vague sync notes), and P2 compliance testing gaps across five specification files.
mcp-scripts-specification.md
SM-01/SM-02/SM-03: implementations MUST sanitize tool stdout through theactions/setup/js/redaction helpers before forwarding to MCP client; secret values from the runner's::add-mask::registry MUST be replaced with"[REDACTED]"T-MS-NEG-001(missingscript/run/py/gofield) andT-MS-NEG-002(input schema referencing undefined type); both added to compliance checklistforecast-specification.md
R-MC-030–R-MC-032: warn when n < 10 ET observations before P10/P90 are reported; simulation still runs (advisory only); includes statistical rationaleR-ERR-040–R-ERR-044: exponential-backoff retry on HTTP 429/403 duringgh apisampling, partial-result fallback withrate_limit_skippedfield (additive optional perR-IMPL-041)specs/forecast-compliance-fixtures/fixture directoryeffective-tokens-specification.md
R-OTL-001–R-OTL-005(MUST usellm.token.effective_total,llm.token.input,llm.token.output,llm.token.base_weighted, etc.)T-ET-006: multi-invocation aggregation with zero-ET leaf nodes; updated test count 12 → 13fuzzy-schedule-specification.md
0x811c9dc5/0x01000193) as the normative hash function with implementation file reference (pkg/parser/schedule_fuzzy_scatter.go)specs/forecast-compliance-fixtures/ (new)
run_summary_minimal.json— minimalRunSummaryfixture with realistic token data to bootstrap §12 compliance testsREADME.md— explains schema mapping, how to rungo test -run "TestForecast|TestMonteCarlo", and guide for adding scenario fixtures