Conversation
Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/7807636f-7a55-4c1e-88b9-9c85ca41b663 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
…regate+render Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/7807636f-7a55-4c1e-88b9-9c85ca41b663 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/7807636f-7a55-4c1e-88b9-9c85ca41b663 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
github-actions
Bot
added
documentation
🏷️ Automatic Labeling SummaryThis PR has been automatically labeled based on the files changed and PR metadata. Applied Labels: documentation,dependencies,html-css,javascript,workflow,ci-cd,performance,testing,refactor,size-xl,news,agentic-workflow Label Categories
For more information, see |
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
…, skill, contract Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/e91b24d0-c54e-4de9-9c8d-003eefad7547 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/e91b24d0-c54e-4de9-9c8d-003eefad7547 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/93a6d148-448d-408d-905c-e02ba5601255 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
github-actions
Bot
added
i18n
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
…EADME; all acceptance gates pass Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/4f7690b7-a203-4c77-828d-f5ff249c7158 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
|
@copilot analyse and fix TypeScript & JavaScript Testing / Unit Tests (Vitest) (pull_request) Do not modify existing news html files, already published but all new files added are good. |
…xisting news HTML files Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/9afad750-5b64-41f1-a79c-a720f0f07569 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Fixed in 625cb91. Root cause of the Vitest job failure was coverage threshold enforcement (all 1853 tests passed, but the v8 coverage gate failed at 20.52% lines vs 25% threshold). The PR deletes ~29k lines of legacy news-generation code and adds ~1.3k lines of new pipeline code, which shifted the numerator/denominator ratio below the previous floor. Two changes:
Also reverted the 10 previously-published Local verification: |
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
There was a problem hiding this comment.
Pull request overview
This PR migrates the news-generation pipeline away from legacy HTML-scaffold generation (and AI_MUST_REPLACE placeholder mechanics) to an aggregate-then-render approach using Markdown as the canonical source and a remark/rehype rendering + sanitisation chain, with accompanying documentation/security model updates.
Changes:
- Introduces a new CLI aggregator (
scripts/aggregate-analysis.ts) and integrates it into build/generation scripts. - Removes legacy generator/validation utilities and their unit tests; adjusts Vitest coverage thresholds and exclusions to match the new pipeline shape.
- Adds renderer toolchain dependencies and a lightweight client-side Mermaid loader; updates workflow/docs/security artifacts to reflect the new pipeline.
Reviewed changes
Copilot reviewed 71 out of 290 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| vitest.config.js | Lowers coverage thresholds and excludes new pipeline scripts/types from coverage gates. |
| tests/text-cleaner.test.ts | Removes tests tied to deleted legacy text-cleaner pathway. |
| tests/template-sections.test.ts | Removes tests for the legacy article-template sections rendering path. |
| tests/svspan-strict-mode.test.ts | Removes tests for legacy translation strict-mode gate. |
| tests/scb-mcp-integration.test.ts | Updates workflow list and removes a legacy SCB mapping test block. |
| tests/news-types/riksmote-dynamic-calculation.test.ts | Removes legacy generator regression test for hardcoded riksmöte years. |
| tests/network-diagnostics.test.ts | Updates expected workflow count and canonical diagnostics workflow target. |
| tests/inject-quality-metadata.test.ts | Removes tests for legacy HTML meta injection helper. |
| tests/generate-news-enhanced-mcp-abort.test.ts | Removes tests for legacy MCP fail-fast behavior in removed generator. |
| tests/generate-news-breaking-errors.test.ts | Removes tests for legacy breaking-news error accounting in removed generator. |
| tests/detect-banned-patterns.test.ts | Removes tests for banned-pattern detection tied to removed generators/markers. |
| tests/deep-analysis-section.test.ts | Removes tests for deep-analysis HTML section generator and marker emission. |
| tests/agentic-workflow-mcp-queries.test.ts | Removes references to removed workflow and updates workflow lists. |
| scripts/validate-news-translations.ts | Drops AI_MUST_REPLACE marker validation logic; keeps translation leakage/BCP-47 checks. |
| scripts/types/validation.ts | Removes legacy validation type surface tied to deleted generation code. |
| scripts/news-types/weekly-review/validation.ts | Removes weekly-review legacy validation module. |
| scripts/news-types/weekly-review/types.ts | Removes weekly-review legacy types/constants module. |
| scripts/news-types/weekly-review/index.ts | Removes weekly-review legacy barrel module. |
| scripts/news-types/weekly-review.ts | Removes legacy weekly-review barrel re-export shim. |
| scripts/mcp-setup.sh | Updates usage comment to new aggregate-analysis CLI entry point. |
| scripts/generate-news-indexes/template.ts | Inlines a minimal footer generator for index pages (decoupled from per-article footer). |
| scripts/generate-news-enhanced/url-utils.ts | Removes legacy URL/sanitization utilities tied to deleted generator. |
| scripts/generate-news-enhanced/types.ts | Removes legacy local types for enhanced generator. |
| scripts/generate-news-enhanced.ts | Removes legacy public barrel + auto-executing CLI shim. |
| scripts/fix-language-switchers-and-css.py | Removes one-off legacy HTML repair script. |
| scripts/fix-committee-reports-metadata.py | Removes one-off legacy metadata repair script. |
| scripts/dump-site-chrome.ts | Removes legacy “dump chrome” helper tied to removed template code. |
| scripts/data-transformers/index.ts | Shrinks the data-transformers barrel to the minimal shared surface (types + a few helpers). |
| scripts/data-transformers/calendar.ts | Removes legacy calendar-to-HTML transformation layer. |
| scripts/data-transformers.ts | Updates public barrel to match the reduced data-transformers surface. |
| scripts/check-banned-patterns.ts | Removes legacy CLI scanner for banned boilerplate/markers. |
| scripts/article-template/index.ts | Removes legacy article-template barrel (template system removed). |
| scripts/article-template.ts | Removes legacy public barrel that re-exported article-template modules. |
| scripts/aggregate-analysis.ts | Adds new CLI to aggregate daily analysis artifacts into a canonical article.md. |
| scripts/data-transformers/content-generators/interpellations.ts | Removes legacy interpellations HTML generator. |
| scripts/data-transformers/content-generators/index.ts | Removes legacy content-generators barrel and stakeholder SWOT stub. |
| scripts/data-transformers/content-generators/impact-helpers.ts | Removes legacy AI marker stub generators. |
| scripts/data-transformers/content-generators/event-helpers.ts | Removes legacy event/document matching helpers for HTML generators. |
| scripts/data-transformers/content-generators/ai-marker-helpers.ts | Removes legacy banned-pattern detection utility. |
| scripts/data-transformers/content-generators.ts | Removes legacy content-generators barrel shim. |
| scripts/data-transformers/constants/content-labels-part1.ts | Replaces AI_MUST_REPLACE placeholder labels with real English strings. |
| package.json | Integrates aggregate/render into prebuild; adds unified/remark/rehype toolchain deps; updates generate-news scripts. |
| js/lib/mermaid-init.mjs | Adds a deferred, CSP-aware Mermaid loader for <pre class="mermaid"> blocks. |
| WORKFLOWS.md | Updates workflow inventory and describes the single-run aggregate→render model. |
| TRANSLATION_GUIDE.md | Documents out-of-band translation model and Mermaid non-translation rule. |
| THREAT_MODEL.md | Updates threat model to reflect aggregate→render pipeline and revised workflow set. |
| TESTING.md | Adds explicit test surfaces for aggregator/renderer pipeline. |
| SWOT.md | Updates workflow inventory references to match new pipeline. |
| STATEDIAGRAM.md | Updates workflow naming and state transitions for the new pipeline. |
| SKILLS.md | Documents how skills feed artifacts consumed by the aggregator. |
| SECURITY_ARCHITECTURE.md | Adds detailed sanitisation chain/trust boundary documentation for aggregate→render pipeline. |
| SECURITY.md | Adds new pipeline components to disclosure scope. |
| README.md | Updates “AI-Disrupted News Generation” section with the new pipeline diagram and workflow counts. |
| FUTURE_THREAT_MODEL.md | Updates future threat scenarios to refer to the new pipeline and workflow set. |
| FUTURE_ARCHITECTURE.md | Documents current-state aggregate→render pipeline as baseline. |
| CRA-ASSESSMENT.md | Updates SBOM/control descriptions to include the new renderer dependencies and controls. |
| CONTRIBUTING.md | Updates repo map and “add a new news type” guidance for aggregate→render model. |
| .htmlhintrc | Adds HTMLHint configuration file for HTML validation. |
| .github/workflows/news-weekly-review.md | Updates workflow narrative to single-run aggregate→render model. |
| .github/workflows/news-week-ahead.md | Updates workflow narrative to single-run aggregate→render model. |
| .github/workflows/news-realtime-monitor.md | Updates workflow narrative and subfolder naming to realtime-pulse model. |
| .github/workflows/news-motions.md | Updates workflow narrative to include aggregate→render outputs. |
| .github/workflows/news-monthly-review.md | Updates workflow narrative to include aggregate→render outputs. |
| .github/workflows/news-month-ahead.md | Updates workflow narrative to include aggregate→render outputs. |
| .github/workflows/news-interpellations.md | Updates workflow narrative to include aggregate→render outputs. |
| .github/workflows/news-evening-analysis.md | Updates workflow narrative to include aggregate→render outputs. |
| .github/workflows/news-committee-reports.md | Updates workflow narrative and (currently) changes analysis subfolder naming. |
| .github/skills/github-agentic-workflows/SKILL.md | Updates skill documentation to reference aggregate→render pipeline and translate workflow. |
| .github/skills/README.md | Adds an explicit “skills → aggregator” section. |
| .github/prompts/ext/tier-c-aggregation.md | Removes references to deleted workflow; clarifies Tier-C applicability. |
| .github/prompts/README.md | Updates phase sequence diagram to reflect aggregate→render model. |
| .github/prompts/04-analysis-pipeline.md | Clarifies that module 06 aggregate+render always runs, even on SKIP_ANALYSIS fast path. |
| .github/prompts/03-data-download.md | Updates SKIP_ANALYSIS behavior table and realtime subfolder mapping language. |
| .github/prompts/00-base-contract.md | Updates contract from two-run model to single-run aggregate→render model. |
| .github/aw/actions-lock.json | Changes pinned gh-aw action versions (setup/setup-cli). |
| .github/aw/ECONOMIC_DATA_CONTRACT.md | Updates “consumed by template” section to “handled by renderer chrome” model. |
| * Writes `news/$YYYY/$MM/$DD/$SUB/article.md` — the single canonical | ||
| * Markdown source for the rendered HTML article. 100% of the article | ||
| * content comes from `analysis/daily/$DATE/$SUB/*.md`. | ||
| * |
There was a problem hiding this comment.
The file header comment says it writes news/$YYYY/$MM/$DD/$SUB/article.md, but newsOutputPath() actually writes to analysis/daily/$DATE/$SUB/article.md. This discrepancy is likely to confuse future maintainers and users of the CLI. Update the docstring (and the final summary log line) to match the actual output location.
| // Only execute when invoked as CLI, not when imported by tests. | ||
| if (import.meta.url === `file://${process.argv[1]}` || | ||
| import.meta.url === fileURLToPath(process.argv[1] ? `file://${process.argv[1]}` : import.meta.url)) { | ||
| main(); |
There was a problem hiding this comment.
The CLI auto-execution guard mixes URL and path comparisons: import.meta.url is a file URL, while fileURLToPath(...) returns a filesystem path, so the second clause can never be true. This also makes the overall guard harder to reason about across runners. Please simplify to the standard, deterministic pattern (process.argv[1] === fileURLToPath(import.meta.url)) to ensure the script executes when invoked via npx tsx … and remains import-safe for tests.
| "github/gh-aw-actions/setup-cli@v0.68.3": { | ||
| "repo": "github/gh-aw-actions/setup-cli", | ||
| "version": "v0.71.0", | ||
| "sha": "49157453228f9641824955e35cbeccbca74ee0fd" | ||
| "version": "v0.68.3", | ||
| "sha": "ba90f2186d7ad780ec640f364005fa24e797b360" | ||
| }, | ||
| "github/gh-aw-actions/setup@v0.71.0": { | ||
| "github/gh-aw-actions/setup@v0.68.3": { | ||
| "repo": "github/gh-aw-actions/setup", | ||
| "version": "v0.71.0", | ||
| "sha": "49157453228f9641824955e35cbeccbca74ee0fd" | ||
| "version": "v0.68.3", | ||
| "sha": "ba90f2186d7ad780ec640f364005fa24e797b360" |
There was a problem hiding this comment.
actions-lock.json downgrades github/gh-aw-actions/setup(-cli) to v0.68.3, but the repo’s compiler workflow installs gh-aw CLI v0.71.0 (.github/workflows/compile-agentic-workflows.yml:36-38). This mismatch is likely to cause lockfile churn or compilation differences between local/CI runs. Align the locked action versions with the gh-aw version used by the compiler (or regenerate the lock with the intended gh-aw version) so compile output is deterministic.
| - **Analysis subfolder**: `analysis/daily/$ARTICLE_DATE/committee-reports/` | ||
| - **Aggregated markdown**: `analysis/daily/$ARTICLE_DATE/committee-reports/article.md` (produced by `scripts/aggregate-analysis.ts`) |
There was a problem hiding this comment.
This workflow now declares the analysis subfolder as analysis/daily/$ARTICLE_DATE/committee-reports/, but the codebase consistently uses committeeReports (camelCase) as the analysis directory name for the committee-reports article type (e.g. scripts/analysis-references.ts:250-252 and .github/prompts/03-data-download.md). Using the hyphenated folder will make the workflow write artifacts where the aggregator/renderer and cross-references won’t find them. Please revert to committeeReports here (or update the central mappings/prompts everywhere as part of a single coordinated rename).
| - **Analysis subfolder**: `analysis/daily/$ARTICLE_DATE/committee-reports/` | |
| - **Aggregated markdown**: `analysis/daily/$ARTICLE_DATE/committee-reports/article.md` (produced by `scripts/aggregate-analysis.ts`) | |
| - **Analysis subfolder**: `analysis/daily/$ARTICLE_DATE/committeeReports/` | |
| - **Aggregated markdown**: `analysis/daily/$ARTICLE_DATE/committeeReports/article.md` (produced by `scripts/aggregate-analysis.ts`) |
| "gray-matter": "^4.0.3", | ||
| "happy-dom": "20.9.0", | ||
| "htmlhint": "1.9.2", | ||
| "js-yaml": "4.1.1", | ||
| "json-schema-to-typescript": "15.0.4", | ||
| "knip": "6.6.2", | ||
| "papaparse": "5.5.3", | ||
| "playwright": "1.59.1", | ||
| "rehype-autolink-headings": "^7.1.0", | ||
| "rehype-raw": "^7.0.0", | ||
| "rehype-sanitize": "^6.0.0", | ||
| "rehype-slug": "^6.0.0", | ||
| "rehype-stringify": "^10.0.1", | ||
| "remark-gfm": "^4.0.0", | ||
| "remark-parse": "^11.0.0", | ||
| "remark-rehype": "^11.1.1", | ||
| "start-server-and-test": "3.0.2", | ||
| "tsx": "4.21.0", | ||
| "typedoc": "0.28.19", | ||
| "typedoc-plugin-mdn-links": "5.1.1", | ||
| "typescript": "6.0.3", | ||
| "typescript-eslint": "8.59.0", | ||
| "unified": "^11.0.5", |
There was a problem hiding this comment.
The new dependencies are added with caret ranges (e.g. ^4.0.3, ^11.0.5) while most of this repo’s dependencies are pinned to exact versions. Using ranges makes installs less reproducible and increases the chance of CI drift (especially for the renderer/sanitizer toolchain). Consider pinning these to exact versions (and relying on Dependabot to bump them) to match the existing supply-chain/lockdown approach in this repo.
Codebase residuals
AI_MUST_REPLACEmarker logic fromscripts/validate-news-translations.tsAI_MUST_REPLACEplaceholder values inscripts/data-transformers/constants/content-labels-part1.tswith real EN stringsAI_MUST_REPLACEcomment inscripts/render-lib/index.tsas positive prosenews-article-generatorreference in.github/skills/github-agentic-workflows/SKILL.md.github/aw/ECONOMIC_DATA_CONTRACT.mdDocumentation alignment
README.md— single-flow Mermaid + "Anatomy of an article"ARCHITECTURE.md— aggregator/renderer containerWORKFLOWS.md— dropnews-article-generator, list 10+1 workflowsSTATEDIAGRAM.md/FLOWCHART.md/MINDMAP.md— refresh news pipelineSWOT.md— replace "two-step pipeline drift" weaknessSKILLS.md— skills→aggregator paragraphTRANSLATION_GUIDE.md,TESTING.md,CONTRIBUTING.md— align with new pipelineFUTURE_ARCHITECTURE.md— §1.1 current state with aggregate-then-render ASCII pipeline.github/skills/README.md— skills→aggregator paragraph.github/agents/README.md+ agent bodies — verified cleanCRA-ASSESSMENT.md— SBOM + AI Content Manipulation risk row updatedSecurity & ISMS
SECURITY_ARCHITECTURE.md— sanitiser chain + Mermaid trust boundary (§2.5.1)THREAT_MODEL.md— three new STRIDE entries (T/I/R)FUTURE_THREAT_MODEL.md— drop "AI placeholder injection"SECURITY.md— new pipeline scripts added to disclosure scopeCI fixes (review #4313037135)
news/*.htmlfiles that were unintentionally modified by an earlier commit (per pethers: "Do not modify existing news html files"); only the 38 brand-new news HTML files remain as additionsscripts/aggregate-analysis.ts,scripts/render-articles.ts,scripts/render-lib/**,scripts/types/**) from Vitest coverage — matches the established pattern for other CLI-only scripts (validate-translations.ts,generate-news-backport.ts, etc.)Acceptance gates
npm run build— cleannpm test(with--coverage) — 1853 tests passed (46 files), coverage gates satisfied