docs: make the front-door claims true, and teach CI to keep them that way - #95
Merged
Conversation
added 3 commits
July 25, 2026 08:05
… model `unknown` is our own stand-in for a row whose model id could not be determined — written into the queue by sync.js and claude-categorizer.js, and coalesced to again at read time by local-api.js. Pricing it turned a missing *attribution* into a missing *price*. That put the placeholder in `unpriced_models`, a field whose whole purpose is to name models needing an entry in curated-overrides.json — where `unknown` can never match. It also logged that exact advice to the console, and made every first lookup schedule an upstream refresh hunting for a price that cannot exist. The field shipped in 0.39.41, so this is what it looked like on first contact. Resolve placeholder ids to a distinct `unattributed` tier before the lookup, so they never reach the negative cache, the tier map, the warning path or the refresh scheduler. A closed set rather than a substring rule: silently un-pricing a real model would recreate the $0 bug this surface exists to expose (#90). A model id that is only whitespace now folds into the `empty` tier instead of leaking into unpriced_models as " ". The dashboard still flags such a row as unpriced — its tokens count toward the total at $0, so the caveat is honest even though the row does not belong in a list of models awaiting a price. Closes #93
… way
The README is well structured, but three of its strongest trust claims were
checkable and failed the check — in a product whose entire pitch is "trust
this number, trust this privacy".
- `tokentracker activate-if-needed`, offered to users whose install is
already broken, has never existed in the CLI's dispatch. It throws.
The working command is `tokentracker init`.
- "local SQLite" described the store; the store is an append-only file at
~/.tokentracker/tracker/queue.jsonl. SQLite is something we only ever
*read*, from tools like Cursor. Reframed as the audit path it actually
is: you can `cat` the whole thing.
- "Nothing ever leaves your laptop" / "no upload path at all" are refutable
in five minutes with lsof — pricing, plan quotas, OAuth refresh and
avatars all make outbound calls. Replaced with the narrower claim that is
actually true ("your usage data never leaves your machine") plus a table
naming every outbound call and why. The weaker claim is the stronger
trust signal, because it survives being checked.
Also: Droid has been wired and tested since sync.js imported it but appeared
nowhere in the README; the hard-coded tool count is now "20+", because a
number in prose has no validator — the same failure mode as the expired
DeepSeek prices. CONTRIBUTING cloned the upstream fork, claimed a stale test
count, and pointed at a README table that no longer exists. `uninstall` does
not remove the launchd agents, which are installed outside the CLI, so
"removes everything" now says so.
Value, not just accuracy: added why you'd use this over each provider's own
billing page, and a section for the macOS menu-bar app and Windows app —
both published as release assets, both invisible in the README, both sitting
at zero downloads.
The durable part is the last commit's worth: scripts/openwiki-check-facts.cjs
already had a check that rejects a documented CLI command which doesn't
exist. It would have caught activate-if-needed the day it was written — it
was only ever pointed at openwiki/, never at the file users actually copy
commands from. It now scans README.md and CONTRIBUTING.md too, while the
completeness half stays scoped to openwiki/ (a front door is not a manifest).
It earned that immediately: it rejected `tokentracker --version` in my own
new prose (also not a real command) and a cautionary example that quoted the
fake command verbatim. There is deliberately no way to silence a finding.
Finally, documents the pricing-diagnostics surface now that #93 is fixed —
including why `unpriced_models` excludes placeholders — and replaces the
publish checklist's hard-coded model names with a query against that surface.
Refs #93
Follow-up review caught this PR committing the mistake it is about. The pricing-tier table asserted "a closed set" and listed six values. The resolution ladder in src/lib/pricing/matcher.js emits eleven. I had read FUZZY_SOURCES and the tiers my own tests exercise, and written the table from those — an unchecked claim in the one change whose thesis is don't ship an unchecked claim. Enumerated from source and regrouped by what each rung means for trusting the number: exactly-resolved, guessed, and not-priced. The four missing rungs were curated:exact-dot, litellm:exact-dot, curated:alias and litellm:strip. Checked whether the omission hid a live defect, since a guessed tier missing from FUZZY_SOURCES would mean fuzzy_priced_models under-reports — the exact invisibility #90 existed to remove. It does not: litellm:strip removes a reasoning-effort suffix (-high, -xhigh, -fast) and then matches the base model exactly, and reasoning effort changes how many tokens you spend, not the rate. curated:alias is a deliberate mapping. Both belong with the exact tiers. FUZZY_SOURCES is correct as written; matcher.js is now cited as the authority so the table cannot drift from it silently. Also: - The scope test wrote a fake command into the tracked README and restored it in a finally block. That covers a thrown assertion but not a killed process or a CI timeout — and the recovery path is `git add README.md`, which would silently re-commit the exact falsehood this PR removes. Now asserts through an exported readRootDocs plus a synthetic collectFindings call. Same two properties proven, nothing tracked is touched. - The publish checklist used `date -v-7d`, which is BSD-only and would fail for a Linux maintainer with nothing in CI to catch it. - Dropped two in-page anchors to emoji headings. GitHub's slugger handles the variation selector in 🛡️ unintuitively and I could not verify the generated id, so the cross-reference is prose. A dead link in the section arguing you should trust these claims is a poor trade for a saved scroll.
Owner
Author
|
Follow-up commit The pricing-tier table asserted "a closed set" and listed six values; I checked whether the omission was hiding a live defect, since a guessed tier missing from Also fixed: the scope test wrote a fake command into the tracked README and restored it in a One thing I left alone, flagged rather than guessed: |
pitimon
pushed a commit
that referenced
this pull request
Jul 25, 2026
Ships #94 and #95. The visible change is that `pricing.unpriced_models` no longer reports the literal placeholder `"unknown"` — 0.39.41 introduced that field and it named a placeholder as though a real model were awaiting a price, which is the one thing the field exists not to do. The rest is documentation, but not cosmetic: the README told users with a broken install to run a command that has never existed, described the store as SQLite when it is an append-only JSONL queue, and claimed nothing ever leaves the machine while making outbound calls for pricing, plan quotas and OAuth refresh. All three are now accurate, with every outbound call named. The durable part is that scripts/openwiki-check-facts.cjs now also scans README.md and CONTRIBUTING.md. It already had a check that rejects a documented CLI command which doesn't exist; it had simply never been pointed at the file users copy commands from. prepublishOnly re-vendored the LiteLLM seed: 2,522 models, no rate changes, only _meta.generated_at moved.
This was referenced Jul 25, 2026
pitimon
added a commit
that referenced
this pull request
Jul 25, 2026
…in CI (#108) * fix(privacy): stop leaking repo names to GitHub, and check the claim in CI Closes #100 and #101. The Projects panel fetched `api.github.com/repos/${project_key}` from the user's browser, and loaded `github.com/${owner}.png` as an <img> in two components. Each request put the name of a repository the user has checked out — private ones included — into a URL sent to a third party. The README said the opposite. It claimed avatars were "fetched server-side so your browser doesn't contact them directly", and listed api.github.com only under provider quota. Both were written by me earlier today, from an audit that grepped `src/` and never `dashboard/src/`. The avatar proxy that claim credits turns out to have no caller in the dashboard at all. Removed the enrichment rather than proxying it. Star counts and owner avatars are decoration on a panel about your own spending, and routing them through the local server would still send the repo name, just from a different process. The durable half is scripts/validate-outbound.cjs, wired into ci:local: - Scans host literals ANYWHERE in src/ and dashboard/src/, not inside fetch( calls. The leak that prompted this was an <img src>; a call-site scan reproduces the original blind spot in code. - Fails on a host the code can reach that outbound-hosts.json does not declare, on a declaration no code references, and on a user-data host missing from the README table. - Enforces seen_in, so a declared host may only be reached from the files that declare it. Without that, re-adding the exact #100 call would pass, because api.github.com is legitimately declared for the header star. - Rejects a fetch() whose target is built at runtime in a file that also names an external host — the hole a literal scan cannot see. It earned its keep on first run: it found `skills.sh`, a ninth host that two rounds of hand-searching had missed, and then that six of my hand-written seen_in lists were wrong. Those are now generated from the scan. The README table is rewritten from the inventory and names 21 hosts where it previously named 12. Also folded in, and called out rather than buried: four links still pointed at the upstream fork, including one `init.js` prints to every user on first run. Same defect class as the CONTRIBUTING clone URL fixed in #95. Two guards on the removal itself: the component tests now assert no remote image renders, and a source-level test asserts no absolute URL appears in ProjectUsagePanel — which fails if someone adds an <img> or <link> rather than a fetch. * fix(validate): the ignore list could exempt more than it named `ignored_prefixes` took URL prefixes and `isIgnored()` reduced each to its host, so the entry `https://github.com/BerriAI` silently exempted every github.com reference in the tree. It was masked only because github.com is also declared, and checked first — remove that declaration and the guard would have gone quiet across a whole domain with nothing to say so. A field that looks narrow and is not is worse than no field, and this one is the validator's own escape hatch. Now `ignored_hosts`, matched on exact host, with the BerriAI entry dropped as redundant. Each remaining entry carries why it is safe; all five were re-verified rather than assumed — `local.tokentracker` is a synthetic project_ref in rollout.js that is never requested, and codex.ai, example.com and evil appear only inside comments. A test asserts a path-shaped entry cannot exempt its host. --------- Co-authored-by: itarun.p <itarun.p@somapait.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #94 — merge that first. This branch contains #94's commit, so the diff here shows it until #94 lands. The dependency is real:
openwiki/local-api.mddocuments theunattributedpricing tier that #94 introduces.Summary
Three README claims were checkable and failed the check — in a product whose whole pitch is "trust this number, trust this privacy":
README:188told broken installs to runtokentracker activate-if-neededUnknown command. Nowtokentracker init.README:99,105"local SQLite"~/.tokentracker/tracker/queue.jsonl. SQLite is only ever read, from tools like Cursor — andREADME:88already said so, so the file contradicted itself.README:53,108,117"Nothing ever leaves your laptop" / "no upload path at all"lsof: LiteLLM pricing, provider quota APIs using your credentials, OAuth refresh, an avatar proxy, an IP-check proxy, and npx itself.The privacy fix is the one worth arguing about. "Your usage data never leaves your machine" is a weaker sentence and a stronger trust signal, because it survives being checked — and it now comes with a table naming every outbound call and why. Same move we made on pricing in #90: prefer a claim that can be verified over one that must be maintained by hand.
Also fixed
sync.jssince it was added, and appears nowhere in the README. Added — and the hard-coded tool count is now "20+", because a number in prose has no validator. That is the same failure mode as the expired DeepSeek prices.mm7894215/TokenTracker(the upstream, read-only for this fork), claimed 96 test files (actual 116), and pointed at a README table that no longer exists.uninstallhas zerolaunchctlreferences, so "removes everything" left aKeepAlivedaemon running. Documented, with the correct script. Scoped honestly: the CLI never installs a LaunchAgent, so only people who ranscripts/install-local-service.share affected.CLAUDE.mdhad the wrong queue path and a stale test count;docs/screenshots/leaderboard.pngwas orphaned by the cloud-era removal (a test now asserts the route is gone).The durable part
scripts/openwiki-check-facts.cjsalready had a check that rejects a documented CLI command which doesn't exist. It would have caughtactivate-if-neededthe day it was written. It was only ever pointed atopenwiki/— never at the file users actually copy commands from.It now scans
README.mdandCONTRIBUTING.mdfor unresolvable references, while the completeness half stays scoped toopenwiki/(a front door is not a manifest). Two tests lock both halves in.It earned its keep immediately: it rejected
tokentracker --versionin my own new prose — also not a real command — and then a cautionary example that quoted the fake command verbatim. There is deliberately no escape hatch; a page that needs to name something nonexistent must describe it rather than quote it.Test plan
npm run ci:local— exit 0, 827/827, fact check 0 findings`tokentracker activate-if-needed`to README makes the checker fail withREADME.md:<line> unknown CLI command— the original bug is now unshippableopenwiki/ missing CLI commandsrc/cli.js, queue path inlocal-api.js:52, outbound hosts by grep,parseDroidIncrementalatsync.js:58,473, 0launchctlrefs inuninstall.js, 116 test files, 2,523 seed pricing entries, release assets viagh release view