Claims that fire on ordinary repositories (v0.5.0) - #25
Merged
Conversation
The verification layer only had two narrow claims (billing webhooks, JWT), so on three realistic open-source repositories dtc verify produced nothing useful on two of them. Honest and useless at the same time. This release fixes the coverage, not the honesty. - New claim route-test-coverage: "HTTP routes are exercised by tests". Routes and tests are the most abundant evidence in nearly every server repo, so this claim fires almost everywhere. It reports the ratio and names the uncovered routes. Attribution is by test imports and route names; e2e specs are excluded because they match by accident; absence of tests is missing evidence, never a contradiction. - New claim admin-authorization: "Administrative routes require an authorization check". A missing authorization signal is WEAK, never CONTRADICTED - global middleware and framework decorators are not detected, and calling a route unprotected when it is not would destroy trust. - New status NOT_APPLICABLE: a repo with no billing code is not "unknown" for a billing claim. UNKNOWN is now reserved for "surface exists, evidence cannot decide". - dtc verify is a report card: findings first (contradictions before all), then which claims do not apply and why. When nothing applies it reports what was scanned, what evidence exists, what would unlock a claim, and that this is a coverage limit rather than a verdict. - dtc verify --list shows applicability for the current repository. - verify_all loads scan evidence once; verify stays sub-second on a repo with 133 routes and thousands of tests. - Results that do not apply are no longer stored: they would pollute freshness and diff impact with claims that have no evidence. JSON is schema_version 2 (all v1 fields unchanged; adds the NOT_APPLICABLE status value). 142 tests (20 new). Version 0.5.0. Investigated and deliberately not built: a secret-handling claim. DevTime hard-denies secret files from scanning, so its evidence cannot exist without breaking the trust model.
The MCP Python SDK released 2.0.0 and renamed its high-level server: mcp.server.fastmcp.FastMCP became mcp.server.MCPServer. Because the extra is declared as mcp>=1.2, every fresh `pipx install "devtime-ei[mcp]"` resolved to 2.0 and failed with ModuleNotFoundError before the server could start. CI caught it on this branch; it was already broken for users on PyPI. Resolve the server class across both SDK generations rather than pinning users to one, and add a regression test asserting the resolved class still exposes tool/list_tools/call_tool/run so a future rename cannot pass silently. Verified on both: MCP 1.28.1 (local) and MCP 2.0.0 (clean venv, wheel install) register all four tools, and verify_claim returns evidence-backed results on both.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
I ran the v0.4 first-run flow against three realistic open-source repos. Two of three produced nothing:
dtc verifydtc verifyThe verification layer was honest and useless at the same time. This fixes the coverage, not the honesty.
What ships
route-test-coverage- "HTTP routes are exercised by tests." Routes and tests are the two most abundant signal kinds in almost every server repo, so this claim fires nearly everywhere. Reports the ratio and names the uncovered routes. Attribution by test imports and route names; e2e specs excluded (they match by accident); absence of tests is missing evidence, never a contradiction.admin-authorization- "Administrative routes require an authorization check." Missing authorization is WEAK, never CONTRADICTED - global middleware and framework decorators aren't detected, and calling an endpoint unprotected when it isn't would destroy the trust the project is built on. The limitation is stated in the output itself.NOT_APPLICABLEstatus - a repo with no billing code isn't "unknown" for a billing claim. UNKNOWN now means only "surface exists, evidence cannot decide".Report card output - findings first (contradictions before everything), then which claims don't apply and why. When nothing applies, it reports what was scanned, what evidence exists, what would unlock a claim, and that this is a coverage limit rather than a verdict - ending with a link to open an issue. The users most likely to have a complaint now land on a screen that invites it.
Design notes
verify_allloads scan evidence once; verify is sub-second on express (133 routes, 1683 tests)schema_version: 2- all v1 fields unchanged, only the new status value addedInvestigated and deliberately not built
A secret-handling claim ("no secrets committed") looked like the highest-value broad claim. The audit killed it: DevTime hard-denies
.env,*.pem,*.keyfrom scanning, so evidence for that claim cannot exist without breaking the trust model. Canon wins over a good feature.Verification
--listapplicabilityVersion 0.5.0. No tag, no publish - awaiting approval.