docs: the web thread, measured (OM2W baseline + A/B verdict + content pipeline)#188
Merged
Conversation
NotASithLord
requested changes
Jul 7, 2026
NotASithLord
left a comment
NotASithLord
left a comment
Owner
There was a problem hiding this comment.
Review pass on the diff. Docs-only change (one new file, docs/benchmarks/2026-07-web-thread.md), no code paths touched — low risk by construction.
Content: sound. Numbers are attributed to their source PRs (#119, #187) with repro commands and $0 integrity-check scripts, matches the house convention of dated/point-in-time benchmark reports rather than hard-coding live state, and the negative A/B result (§2) is reported honestly rather than omitted — including the retracted recording-artifact detail, which is exactly the kind of self-correction this repo's docs conventions ask for. No factual claims here contradict what's in #119/#187's own descriptions or docs/security conventions. CI green.
Blocking: missing DCO sign-off. The single commit (ae896ae3) has no Signed-off-by trailer. Same gap that blocked #170's merge and is called out again on #189 (companion PR from the same session) — please amend with git commit -s before merge.
No other concerns — once sign-off is added this is mergeable as-is.
Generated by Claude Code
…ict, content-pipeline numbers Signed-off-by: Jonathan Bursztyn <jobur93@gmail.com>
jonybur
force-pushed
the
docs/web-thread-report
branch
from
ae896ae to
a7b8176
Compare
Collaborator
Author
|
Signed — the commit now carries the DCO trailer ( |
NotASithLord
approved these changes
Jul 8, 2026
NotASithLord
left a comment
NotASithLord
left a comment
Owner
There was a problem hiding this comment.
Dropping the DCO block — the sign-off checkbox is being removed from the PR template (history was 78% unsigned; the convention was never enforced). Docs-only, CI green, approving on the merits.
Generated by Claude Code
5 tasks
NotASithLord
pushed a commit
that referenced
this pull request
Jul 8, 2026
A skeptical, adversarially-verified review of the four just-merged PRs surfaced a set of correctness, integration, doc, and test-hygiene defects. Each fix below survived a 2–3 vote adversarial verification pass. Correctness - manifests.js: the `research` / `browse-only` presets allowed a spill producer (fetch_url, read_page mode:'content') but omitted read_web_cache, so a spilled body's trusted paging footer instructed a tool the manifest gate then refused — the spilled tail unreachable, turns burned on guaranteed-refused calls. Add read_web_cache to both presets (read-only, local cache, no new authority). - read-page.js: mode:'content' clipped the raw DOM at 2MB but reported truncated:false, silently dropping the tail. Surface an explicit htmlTruncated signal (distinct from the paging `truncated` flag). - read-web-cache.js: the evicted-entry hint said "re-fetch the URL", which for a read_page spill steers the actor to fetch_url a page it already rendered (losing the post-JS DOM). Make the recovery hint source-aware. Provider integration (#170 rebase + adjacent) - error-classify.js: Z.ai signals out-of-credit as "insufficient balance … recharge" on HTTP 429; the needles missed it, so a spent GLM account got 3 pointless retries and no failover. Add the needles (+ regression test). - pricing.js: glm-4.6 was priced identically to the glm-5.2 flagship ($1.4/$4.4), a ~3x overestimate that inflated cost and tripped spend limits early. Correct to its published tier ($0.43/$1.74) (+ regression test). - openai.js: the stream parser was called without a provider label, so in-stream errors on OpenAI chats were attributed to "openrouter". Pass { provider: 'openai' }. Docs / classification - CLAUDE.md, README.md, peerd-provider/index.js: provider inventory said OpenAI "doesn't exist yet" and omitted OpenAI + Z.ai GLM; both ship. Update (and defer to registry.js per the no-hard-coded-inventories rule). - idb.js: web_extract_cache was labeled "fetched public bytes"; read_page content mode can store a logged-in tab's rendered DOM. Correct the comment. - error-display.js: usage-limit copy hard-coded "Anthropic / OpenRouter billing"; genericize (the error already names the provider dynamically). - docs/benchmarks/2026-07-web-thread.md: reconcile the failure-taxonomy count (categories overlap) and the 6/29 code-arm denominator (one run did not complete scoring). Test hygiene - spawn.test.ts: the "strips EVERY capability" test passed vacuously (the fixture omitted webCache). Complete the fixture, keep the universal claim honest, and add the read_page→webCache grant assertion #189 introduced. - read-page-content.test.ts: pin the grab-failure fail-open path the header claimed but never exercised, plus the new htmlTruncated signal. NOT fixed here (flagged for a deliberate follow-up): unpriced runner models (glm-4.5-air, OpenRouter's default haiku) render $0.00 and bypass the spend limit — the correct fix is a cost-path change (thread `estimated` through the tally; fail-closed on unpriced under a set limit) affecting multiple providers, not an obvious one-liner. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L8EYHtUpzTX6YGfEXPRPKF
5 tasks
NotASithLord
added a commit
that referenced
this pull request
Jul 8, 2026
A skeptical, adversarially-verified review of the four just-merged PRs surfaced a set of correctness, integration, doc, and test-hygiene defects. Each fix below survived a 2–3 vote adversarial verification pass. Correctness - manifests.js: the `research` / `browse-only` presets allowed a spill producer (fetch_url, read_page mode:'content') but omitted read_web_cache, so a spilled body's trusted paging footer instructed a tool the manifest gate then refused — the spilled tail unreachable, turns burned on guaranteed-refused calls. Add read_web_cache to both presets (read-only, local cache, no new authority). - read-page.js: mode:'content' clipped the raw DOM at 2MB but reported truncated:false, silently dropping the tail. Surface an explicit htmlTruncated signal (distinct from the paging `truncated` flag). - read-web-cache.js: the evicted-entry hint said "re-fetch the URL", which for a read_page spill steers the actor to fetch_url a page it already rendered (losing the post-JS DOM). Make the recovery hint source-aware. Provider integration (#170 rebase + adjacent) - error-classify.js: Z.ai signals out-of-credit as "insufficient balance … recharge" on HTTP 429; the needles missed it, so a spent GLM account got 3 pointless retries and no failover. Add the needles (+ regression test). - pricing.js: glm-4.6 was priced identically to the glm-5.2 flagship ($1.4/$4.4), a ~3x overestimate that inflated cost and tripped spend limits early. Correct to its published tier ($0.43/$1.74) (+ regression test). - openai.js: the stream parser was called without a provider label, so in-stream errors on OpenAI chats were attributed to "openrouter". Pass { provider: 'openai' }. Docs / classification - CLAUDE.md, README.md, peerd-provider/index.js: provider inventory said OpenAI "doesn't exist yet" and omitted OpenAI + Z.ai GLM; both ship. Update (and defer to registry.js per the no-hard-coded-inventories rule). - idb.js: web_extract_cache was labeled "fetched public bytes"; read_page content mode can store a logged-in tab's rendered DOM. Correct the comment. - error-display.js: usage-limit copy hard-coded "Anthropic / OpenRouter billing"; genericize (the error already names the provider dynamically). - docs/benchmarks/2026-07-web-thread.md: reconcile the failure-taxonomy count (categories overlap) and the 6/29 code-arm denominator (one run did not complete scoring). Test hygiene - spawn.test.ts: the "strips EVERY capability" test passed vacuously (the fixture omitted webCache). Complete the fixture, keep the universal claim honest, and add the read_page→webCache grant assertion #189 introduced. - read-page-content.test.ts: pin the grab-failure fail-open path the header claimed but never exercised, plus the new htmlTruncated signal. NOT fixed here (flagged for a deliberate follow-up): unpriced runner models (glm-4.5-air, OpenRouter's default haiku) render $0.00 and bypass the spend limit — the correct fix is a cost-path change (thread `estimated` through the tally; fail-closed on unpriced under a set limit) affecting multiple providers, not an obvious one-liner. Claude-Session: https://claude.ai/code/session_01L8EYHtUpzTX6YGfEXPRPKF Co-authored-by: Claude <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A dated, self-contained benchmark report for the web thread — written so the story is liftable straight into a post:
Repro commands + $0 integrity checks included. Point-in-time report (dated), not live state — per the docs-defer-to-code rule.
🤖 Generated with Claude Code