Enforce methodology-reflection contract: 9 required sections + gate/test validation for KJ coverage, posterior confidence, and re-run schema#2496
Merged
Conversation
Contributor
🏷️ Automatic Labeling SummaryThis PR has been automatically labeled based on the files changed and PR metadata. Applied Labels: size-xs Label Categories
For more information, see |
Contributor
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
8 tasks
Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/490ae0e9-9598-4271-aabd-69926c634300 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Enforce 100% KJ coverage and unify templates
Enforce methodology-reflection contract: 9 required sections + gate/test validation for KJ coverage, posterior confidence, and re-run schema
May 15, 2026
github-actions
Bot
added
documentation
Contributor
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
Contributor
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the agentic analysis gate and templates around methodology-reflection.md, making previously informal run-audit requirements explicit across templates, prompt documentation, gate logic, and tests.
Changes:
- Expands methodology-reflection requirements to nine required sections.
- Adds related template guidance for KJ coverage, posterior confidence, and Lagrådet/Statskontoret/SKR tracking.
- Updates TypeScript and prompt-level gate checks plus unit tests for the new methodology-reflection contract.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
tests/agentic-analysis-gate.test.ts |
Adds shared valid methodology-reflection fixture and updates Family C tests. |
scripts/agentic/analysis-gate.ts |
Replaces loose methodology-reflection check with nine required-section predicates. |
analysis/templates/methodology-reflection.md |
Adds gate-enforced required-section contract and supporting tables. |
analysis/templates/lagradet-tracking.md |
Adds reusable tracking fragment for legal/implementation signals. |
analysis/templates/intelligence-assessment.md |
Adds posterior confidence distribution table guidance. |
analysis/templates/devils-advocate.md |
Adds required KJ coverage matrix guidance. |
.github/prompts/05-analysis-gate.md |
Updates analysis gate prompt and inline shell checks for the new contract. |
.github/prompts/04-analysis-pipeline.md |
Updates artifact catalogue expectations for methodology-reflection. |
|
|
||
| ## 🎯 Key Judgment Coverage Matrix (Required) | ||
|
|
||
| > **[REQUIRED]** Before writing hypotheses, list **every** KJ from `intelligence-assessment.md` and map it to at least one challenge row. Gate expectation: **100% KJ coverage**. |
Comment on lines
+797
to
+835
| const requiredSectionChecks: Array<{ label: string; present: boolean }> = [ | ||
| { label: 'ICD 203 audit checklist', present: /ICD\s+203/i.test(content) }, | ||
| { | ||
| label: "Devil's-Advocate KJ coverage matrix", | ||
| present: /Devil'?s[-\s]Advocate.*(KJ|Key\s+Judgment).*Coverage/i.test(content), | ||
| }, | ||
| { | ||
| label: 'Confidence distribution with posterior', | ||
| present: /Confidence\s+Distribution/i.test(content) && /Posterior/i.test(content), | ||
| }, | ||
| { | ||
| label: 'Lagrådet / Statskontoret / SKR tracking', | ||
| present: lower.includes('lagrådet') && | ||
| lower.includes('statskontoret') && | ||
| (lower.includes('skr') || lower.includes('sveriges kommuner')), | ||
| }, | ||
| { | ||
| label: 'Sibling-folder ingestion record', | ||
| present: /Sibling[-\s]?Folder\s+Ingestion/i.test(content), | ||
| }, | ||
| { | ||
| label: 'Re-run log unified schema', | ||
| present: hasReRunColumns, | ||
| }, | ||
| { | ||
| label: 'Banned-phrase audit grid', | ||
| present: /Banned[-\s]?Phrase\s+Audit/i.test(content), | ||
| }, | ||
| { | ||
| label: 'Pass 1 to Pass 2 delta table', | ||
| present: lower.includes('pass 1') && lower.includes('pass 2') && lower.includes('delta'), | ||
| }, | ||
| { | ||
| label: 'Improvement opportunities linked to PIR roll-forward', | ||
| present: lower.includes('improvement opportunities') && | ||
| lower.includes('pir') && | ||
| lower.includes('roll-forward'), | ||
| }, | ||
| ]; |
Comment on lines
+233
to
+238
| grep -qiE 'Banned[-[:space:]]?Phrase[[:space:]]+Audit' "$REF" \ | ||
| || { echo "❌ methodology-reflection.md: missing banned-phrase audit grid"; FAIL=1; } | ||
| grep -qiE 'Pass[[:space:]]*1.*Pass[[:space:]]*2.*Delta' "$REF" \ | ||
| || { echo "❌ methodology-reflection.md: missing Pass 1→Pass 2 delta table"; FAIL=1; } | ||
| grep -qiE 'Improvement[[:space:]]+Opportunities.*PIR[[:space:]]+Roll[-[:space:]]?Forward' "$REF" \ | ||
| || { echo "❌ methodology-reflection.md: missing improvement opportunities to PIR roll-forward section"; FAIL=1; } |
|
|
||
| --- | ||
|
|
||
| ## 🎯 Devil's-Advocate Key Judgment Coverage Matrix |
Comment on lines
+140
to
+144
| | Track | Applicability | Status | Latest evidence (URL + date) | Next check | | ||
| |-------|:-------------:|--------|-------------------------------|------------| | ||
| | Lagrådet yttrande | Y/N | `not applicable / referral pending / yttrande published` | `[REQUIRED]` | `[REQUIRED]` | | ||
| | Statskontoret implementation signal | Y/N | `found / none found / not triggered` | `[REQUIRED]` | `[REQUIRED]` | | ||
| | SKR operational impact signal | Y/N | `found / none found / not triggered` | `[REQUIRED]` | `[REQUIRED]` | |
Comment on lines
+797
to
+801
| const requiredSectionChecks: Array<{ label: string; present: boolean }> = [ | ||
| { label: 'ICD 203 audit checklist', present: /ICD\s+203/i.test(content) }, | ||
| { | ||
| label: "Devil's-Advocate KJ coverage matrix", | ||
| present: /Devil'?s[-\s]Advocate.*(KJ|Key\s+Judgment).*Coverage/i.test(content), |
Comment on lines
+803
to
+806
| { | ||
| label: 'Confidence distribution with posterior', | ||
| present: /Confidence\s+Distribution/i.test(content) && /Posterior/i.test(content), | ||
| }, |
| grep -qiE "Devil'?s[-[:space:]]Advocate.*(KJ|Key[[:space:]]+Judgment).*Coverage" "$REF" \ | ||
| || { echo "❌ methodology-reflection.md: missing Devil's-Advocate KJ-coverage matrix"; FAIL=1; } | ||
| grep -qiE 'Confidence[[:space:]]+Distribution' "$REF" || { echo "❌ methodology-reflection.md: missing confidence distribution section"; FAIL=1; } | ||
| grep -qiE 'Posterior' "$REF" || { echo "❌ methodology-reflection.md: missing Posterior column/marker"; FAIL=1; } |
Member
|
@copilot apply changes based on the comments in this thread |
Contributor
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
…nt-quality checks Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/14eeb3d8-7441-4b8b-9272-2344f375b52f Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Contributor
Author
Addressed all 8 review-thread findings in 8019abe:
|
github-actions
Bot
added
translation
Contributor
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
Contributor
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/agentic/analysis-gate.tsto##markdown headings (with optional leading emoji) instead of loose phrase-anywhere matchesmethodology-reflection.md: no❌rows and noOPENstatuses on KJ rows (100% coverage enforced)[REQUIRED]placeholder, no empty cell, locates Posterior column from header)checkDevilsAdvocateto require## Key Judgment Coverage Matrixheading with 100% KJ coverage (no❌rows).github/prompts/05-analysis-gate.md(including awk-based posterior column validation)methodology-reflection.mdtemplate: contract now matches actual emoji headings used in the template; addImpact notecolumn to Lagrådet table to match canonicallagradet-tracking.mdfragment05-analysis-gate.mdto document the new content-quality requirements