Skip to content

Add a test that keeps docs/DEPS.md in parity with the manifests #5708

Description

@atomantic

Problemdocs/DEPS.md calls itself a "Living reference of every third-party dependency in PortOS" and is the artifact a reviewer consults to answer "why is this package here, and is it still justified?". Nothing enforces that claim, and it has already drifted: playwright-core — a server runtime dependency added on 2026-08-31, after the last audit — has no row at all, so the newest dependency in the repo is the one with no recorded justification. The document also misfiles pm2 and records a stale version for @biomejs/biome. Every dependency-related finding in this audit ends with "and update DEPS.md", which is precisely the manual step that keeps failing; a test makes the next omission fail at the commit that introduces it instead of at the next audit, months later.

Evidence — The document's own scope claim, docs/DEPS.md:1-5:

# Dependency Audit (DEPS.md)

Living reference of every third-party dependency in PortOS, why it's kept, and what the current verdict is. Updated by `/do:depfree` runs.

**Last audited:** 2026-08-04 ...

Measured parity gap — for each manifest, dependency names with no backticked mention anywhere in docs/DEPS.md:

root       missing from DEPS.md: []
server     missing from DEPS.md: ["playwright-core"]
client     missing from DEPS.md: []
autofixer  missing from DEPS.md: []

playwright-core@1.62.1 is real and load-bearing — server/package.json:34, imported at server/services/fableLoom/falVideoAutomation.js:13 (import { chromium } from 'playwright-core';) with six further import('playwright-core') type references in the same file and a vi.mock('playwright-core', ...) in server/services/fableLoom/falVideoAutomation.test.js:15. It entered the tree in the 2026-08-31 commit "automate fal.ai scene video production", 27 days after docs/DEPS.md's stated last-audit date.

The reverse direction is clean once intentional history is excluded — DEPS.md rows naming packages absent from every manifest are exactly the five deliberately-recorded removals (eslint, @eslint/js, @eslint-react/eslint-plugin, eslint-plugin-react-hooks, typescript), all marked REMOVED in the Verdict column. So the test can allow rows whose verdict is REMOVED/REPLACED and still be meaningful.

Two smaller drifts in the same document:

  • docs/DEPS.md:22 heads the pm2 row **Root devDeps**, but package.json:49-51 places it in "dependencies". pm2 is a runtime process manager invoked by npm start; dependencies is correct and the heading is wrong.
  • docs/DEPS.md:98 states "replaced eslint with @biomejs/biome@2.5.7"; client/package.json:40 pins 2.5.11.

Plan

  1. Create docs/deps-doc.test.js. Decision: colocate the test with the document it guards rather than in server/, and rely on the server Vitest config already globbing sibling directories (AGENTS.md:16 — "cd server && npm test ALSO globs ../scripts, ../lib, ../autofixer"). The include array at server/vitest.config.js:56-64 currently lists '**/*.test.js', '../scripts/**/*.test.js', '../lib/**/*.test.js' and '../autofixer/**/*.test.js' — no docs glob — so add '../docs/**/*.test.js' to it, with a one-line comment matching the style of the autofixer entry above it. Do not relocate the test into server/: a DEPS.md guard belongs next to DEPS.md.
  2. In that test, discover manifests via discoverWorkspaces() + workspaceDir() from scripts/trusted-rebuilds.js (the repo's existing single source for "what is a workspace"), so a fifth workspace is covered automatically instead of needing a hardcoded list edit.
  3. docs/DEPS.md — add the missing playwright-core row to the Server deps section: | \playwright-core` | 1 | KEEP | `server/services/fableLoom/falVideoAutomation.js` — fal.ai video automation | Browser automation without the full `playwright` package's bundled browser download; the browser is provisioned separately by `npm run setup:browser` |`.
  4. docs/DEPS.md:22 — change the **Root devDeps** section heading to **Root deps**.
  5. docs/DEPS.md:98 — correct @biomejs/biome@2.5.7 to @biomejs/biome@2.5.11, or reword to drop the version (the Quick Reference row is the version's home). Decision: drop the version from the prose and let the manifest be the single source, so this line cannot drift again.
  6. docs/DEPS.md:5 — update the **Last audited:** line with today's date and a one-clause summary of this audit pass.

Tests

  • docs/deps-doc.test.js, it('documents every dependency declared in every workspace manifest') — for each discovered workspace with a package.json, assert every dependencies/devDependencies key appears backticked in docs/DEPS.md. This uniquely catches the playwright-core class of miss: a dependency added between audits with no recorded justification. No existing test reads docs/DEPS.md at all.
  • it('names no package that no manifest declares, unless the row records a removal') — parse the Quick Reference table rows, and for each package name absent from every manifest, assert the row's Verdict column contains REMOVED or REPLACED. This uniquely catches a dependency that was dropped from a manifest while its KEEP row lingered — a reviewer reading that row would conclude the package is installed when it is not.
  • it('scans a non-empty set of manifests and table rows') — assert at least 4 manifests and at least 40 table rows were examined. Without it, a path or regex change that silently matches nothing passes both assertions above as clean; this is the same non-vacuity guard already used at scripts/trusted-rebuilds.test.js:200.
  • Do not assert on version strings in the document — DEPS.md deliberately does not carry a version for most rows, and asserting on the few that do would make every Dependabot bump a doc-edit chore.

Acceptance criteria

  • docs/deps-doc.test.js exists and all three tests pass.
  • The parity test fails if the playwright-core row is deleted from docs/DEPS.md.
  • The non-vacuity test fails if the manifest-discovery path is broken.
  • docs/DEPS.md has a playwright-core row, heads the root section **Root deps**, and no longer names a stale Biome version.
  • docs/DEPS.md **Last audited:** reflects this pass.
  • cd server && npm test picks up and passes the new test file.

Out of scope — Do not add the autofixer/browser workspace sections to docs/DEPS.md here (that is deps-01); this PR adds the test and fixes only the playwright-core, pm2-heading, and Biome-version drifts. Do not reclassify any package's tier. Do not rewrite the Detailed Findings or Install-Script Policy sections.


Filed by a /do:better --scan-only --issues audit (2026-09-01). Category: deps · Severity: low · Files: docs/DEPS.md:5, docs/DEPS.md:33, server/package.json:34, server/dependency-overrides.test.js:1

All labels already exist in the repo; do NOT create labels. Never add planner:* labels.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:devtoolsDevtools/workspace/code-review surfacesdepsDependency Freedomeffort:mediumEffort: mediummodel:mediumModel size: mediumplanTracked by /do:replanplanner:fable-5.1Plan authored by the Fable 5.1 modelseverity:lowtestsTest suite / test infrastructure

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions