Skip to content

results: rerun the page view and Claude suites on oc 0.5.1 - #3

Merged
only-cli merged 1 commit into
mainfrom
results/rerun-0.5.1
Sep 2, 2026
Merged

results: rerun the page view and Claude suites on oc 0.5.1#3
only-cli merged 1 commit into
mainfrom
results/rerun-0.5.1

Conversation

@only-cli

@only-cli only-cli commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Reruns the benchmarks against the published oc 0.5.1 on 2026-09-02 and updates the README and TASKS.md to match.

What ran:

  • Page view suite, all eleven adapters, OC_BIN="npx -y @only-cli/oc@0.5.1", lynx 2.9.0 present.
  • Browse suite on Claude Code (claude-sonnet-5), 65 sessions.
  • Wikipedia and language docs suites on Claude Code, 15 and 33 sessions.

What changed in the numbers:

  • oc reads 13 of 15 pages for 8,971 tokens. Reddit now redirects logged-out readers to a login wall, and every other adapter returned that wall or a 403 block page as a success; oc is the only one that reports a failure. On the twelve pages both read, raw HTML costs 125x what oc does.
  • Yahoo Finance now refuses plain fetch outright; oc reads it at 452 tokens.
  • Browse: oc 13/13 at 1,748,589 tokens and $0.81, lynx 1,359,717 and $0.70, Jina Reader 1,456,559, Playwright MCP 2,291,624, raw curl 2,683,048 with one failure. The Azure page is still oc's one bad row (12 turns, 309,076 tokens) because the find/read numbering mismatch is still present in 0.5.1.
  • Wiki: oc 5/5 at $0.23 against WebFetch $0.35 and WebSearch $0.45, all correct.
  • Docs: oc 11/11 at $0.56, WebFetch 10/11 at $0.71 (cppreference 403 again), WebSearch 11/11 at $0.85.

Not rerun: the three Codex suites. codex exec fails with "Your access token could not be refreshed" on this machine, so the Codex tables and their dates are unchanged and labelled as not yet rerun.

Page view suite against the published 0.5.1, then the browse, wiki and
docs suites on claude-sonnet-5, 113 sessions in all, with lynx 2.9.0
present. Reddit now sends logged-out readers to a login wall, so both
Reddit pages and both Reddit tasks are blocks for every tool; oc is the
only page view adapter that reports that as a failure rather than
returning the wall as content. The Codex tables are unchanged: codex
exec could not refresh its sign-in on this machine, so those suites
wait for a fresh login.
@only-cli
only-cli merged commit 1eff3b2 into main Sep 2, 2026
@only-cli
only-cli deleted the results/rerun-0.5.1 branch September 2, 2026 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant