Skip to content

Claims that fire on ordinary repositories (v0.5.0) - #25

Merged
Shakargy merged 2 commits into
mainfrom
v0.5.0-claims-that-fire
Aug 12, 2026
Merged

Claims that fire on ordinary repositories (v0.5.0)#25
Shakargy merged 2 commits into
mainfrom
v0.5.0-claims-that-fire

Conversation

@Shakargy

Copy link
Copy Markdown
Owner

Why

I ran the v0.4 first-run flow against three realistic open-source repos. Two of three produced nothing:

Repo v0.4 dtc verify v0.5 dtc verify
FastAPI full-stack template UNKNOWN, SUPPORTED SUPPORTED (JWT) + WEAK: 6 of 15 routes have a referencing test
expressjs/express UNKNOWN, UNKNOWN WEAK: 59 of 133 routes have a referencing test
gin-gonic/gin (Go) UNKNOWN, UNKNOWN Explains that no claim applies, what was scanned, and what would unlock one

The verification layer was honest and useless at the same time. This fixes the coverage, not the honesty.

What ships

route-test-coverage - "HTTP routes are exercised by tests." Routes and tests are the two most abundant signal kinds in almost every server repo, so this claim fires nearly everywhere. Reports the ratio and names the uncovered routes. Attribution by test imports and route names; e2e specs excluded (they match by accident); absence of tests is missing evidence, never a contradiction.

admin-authorization - "Administrative routes require an authorization check." Missing authorization is WEAK, never CONTRADICTED - global middleware and framework decorators aren't detected, and calling an endpoint unprotected when it isn't would destroy the trust the project is built on. The limitation is stated in the output itself.

NOT_APPLICABLE status - a repo with no billing code isn't "unknown" for a billing claim. UNKNOWN now means only "surface exists, evidence cannot decide".

Report card output - findings first (contradictions before everything), then which claims don't apply and why. When nothing applies, it reports what was scanned, what evidence exists, what would unlock a claim, and that this is a coverage limit rather than a verdict - ending with a link to open an issue. The users most likely to have a complaint now land on a screen that invites it.

Design notes

  • verify_all loads scan evidence once; verify is sub-second on express (133 routes, 1683 tests)
  • NOT_APPLICABLE results are not stored - they'd pollute freshness and diff impact with claims that have no evidence
  • Evidence is capped at 25 routes with the truncation disclosed in limitations, so bounded output never silently misrepresents freshness coverage
  • JSON is schema_version: 2 - all v1 fields unchanged, only the new status value added

Investigated and deliberately not built

A secret-handling claim ("no secrets committed") looked like the highest-value broad claim. The audit killed it: DevTime hard-denies .env, *.pem, *.key from scanning, so evidence for that claim cannot exist without breaking the trust model. Canon wins over a good feature.

Verification

  • 142 tests passing (20 new): all statuses per claim, the WEAK-not-CONTRADICTED honesty rule, e2e exclusion, contradiction-first ordering, NOT_APPLICABLE not persisted, report-card and dead-end output, schema v2, --list applicability
  • build + twine clean

Version 0.5.0. No tag, no publish - awaiting approval.

The verification layer only had two narrow claims (billing webhooks, JWT), so
on three realistic open-source repositories dtc verify produced nothing useful
on two of them. Honest and useless at the same time. This release fixes the
coverage, not the honesty.

- New claim route-test-coverage: "HTTP routes are exercised by tests". Routes
  and tests are the most abundant evidence in nearly every server repo, so this
  claim fires almost everywhere. It reports the ratio and names the uncovered
  routes. Attribution is by test imports and route names; e2e specs are
  excluded because they match by accident; absence of tests is missing
  evidence, never a contradiction.
- New claim admin-authorization: "Administrative routes require an
  authorization check". A missing authorization signal is WEAK, never
  CONTRADICTED - global middleware and framework decorators are not detected,
  and calling a route unprotected when it is not would destroy trust.
- New status NOT_APPLICABLE: a repo with no billing code is not "unknown" for a
  billing claim. UNKNOWN is now reserved for "surface exists, evidence cannot
  decide".
- dtc verify is a report card: findings first (contradictions before all), then
  which claims do not apply and why. When nothing applies it reports what was
  scanned, what evidence exists, what would unlock a claim, and that this is a
  coverage limit rather than a verdict.
- dtc verify --list shows applicability for the current repository.
- verify_all loads scan evidence once; verify stays sub-second on a repo with
  133 routes and thousands of tests.
- Results that do not apply are no longer stored: they would pollute freshness
  and diff impact with claims that have no evidence.

JSON is schema_version 2 (all v1 fields unchanged; adds the NOT_APPLICABLE
status value). 142 tests (20 new). Version 0.5.0.

Investigated and deliberately not built: a secret-handling claim. DevTime
hard-denies secret files from scanning, so its evidence cannot exist without
breaking the trust model.
The MCP Python SDK released 2.0.0 and renamed its high-level server:
mcp.server.fastmcp.FastMCP became mcp.server.MCPServer. Because the extra is
declared as mcp>=1.2, every fresh `pipx install "devtime-ei[mcp]"` resolved to
2.0 and failed with ModuleNotFoundError before the server could start. CI
caught it on this branch; it was already broken for users on PyPI.

Resolve the server class across both SDK generations rather than pinning users
to one, and add a regression test asserting the resolved class still exposes
tool/list_tools/call_tool/run so a future rename cannot pass silently.

Verified on both: MCP 1.28.1 (local) and MCP 2.0.0 (clean venv, wheel install)
register all four tools, and verify_claim returns evidence-backed results on
both.
@Shakargy
Shakargy merged commit 2a86537 into main Aug 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant