browser-worker

An advanced browser search agent. Give it a research question; it searches the web, fetches the real pages, extracts the actual article text, drops duplicates, ranks what is left with a transparent scoring function, and emits a structured result list.

No API keys. No LLM in the loop — the ranking is arithmetic you can read. HTTP only by default, with your own Brave available on demand for pages that need JavaScript (--render auto), and a token budget for when the output is going to a model.

query ─▶ search ─▶ fetch ─────▶ extract ─▶ dedupe ─▶ rank ─▶ json / csv / md
         (DDG)    (async,    │  (title,    (URL +    (5 weighted   agent / brief
                  robots,    │   date,      simhash)  signals)     + --budget
                  retry,     │   text,
                  cached)    │   links)
                             └─ Brave (CDP) for 403s and JS-only pages

Install

Python 3.12 is required (3.13+ untested, 3.11 and below will reject the type syntax in a few annotations).

py -3.12 -m venv .venv
.\.venv\Scripts\pip.exe install -r requirements.txt

Usage

.\.venv\Scripts\python.exe -m browser_worker "how do vector databases index embeddings" --out results.json

Real output from that exact command lives in examples/ — results.json, results.csv, results.md, and results.agent.json (the same run at an 1200-token budget: 30.5 KB down to 5.7 KB).

Every flag

Flag	Default	What it does
query	—	positional; the research question
-n, --max-results	20	how many search hits to consider before fetching
--engine	duckduckgo	search backend; repeatable, one --engine per backend
--top	0	keep only the top K after ranking; 0 keeps everything
-c, --concurrency	5	maximum simultaneous fetches
--timeout	15.0	per-request timeout, seconds
--retries	3	attempts per URL before giving up
--delay	1.0	minimum seconds between two hits on the same domain
--ignore-robots	off	skip robots.txt. Impolite; off by default
--cache	per-user path	SQLite cache path (%LOCALAPPDATA%\browser-worker\cache.db on Windows)
--cache-ttl	86400	cache entry lifetime in seconds; -1 never expires
--no-cache	off	disable the cache entirely
--format	json	json, csv, md, agent (compact JSON), or brief (cheapest)
-o, --out	stdout	write to this file instead of stdout
--similarity	0.90	near-duplicate threshold, 0–1; lower drops more
--budget	0	spend at most ~N tokens on page text (0 = unbudgeted)
--text-chars	2000	per-result cap on raw text before any budget applies
--no-text	off	omit page text entirely — titles, URLs and scores only
--lean	off	drop score_breakdown, outbound_links and other agent-useless fields
-v, --verbose	off	-v info, -vv debug (to stderr)
--version	—	print version and exit

Every run prints an estimated token count to stderr, so the cost is visible without guessing.

Exit code is 0 when at least one result was produced, 1 when none were.

Use it as a Claude Code skill

skill/SKILL.md packages this as a skill, so Claude Code reaches for it on its own when a question needs current or verifiable information — and you can invoke it directly with /browser-worker <question>.

Install it for every project:

mkdir "$env:USERPROFILE\.claude\skills\browser-worker"
copy skill\SKILL.md "$env:USERPROFILE\.claude\skills\browser-worker\SKILL.md"

Or scope it to one repo by copying to .claude/skills/browser-worker/SKILL.md there.

The skill tells the model to default to --format brief --budget 1200 --top 5, how to read the output, when to reach for --render auto, and — importantly — to say so when results are thin rather than dress them up.

For this to work from any directory, install the package itself rather than relying on the current directory being on sys.path:

.\.venv\Scripts\pip.exe install -e .

Token budget — feeding this to a model

The default JSON is written for a human opening a file. An agent pays for every character, and most of that payload buys nothing: text is the first 2000 characters of the page, so the tokens go on navigation, cookie notices and marketing intros rather than on the part that answers the question.

--format agent and --budget fix both halves of that.

Measured on the real examples/results.json (8 results from the live run above), reproducible with python -m browser_worker.compress:

Mode	Tokens	Saving
baseline (--format json)	~9093	—
--lean	~5222	43%
--format agent --budget 1200	~1639	82%
--format brief --budget 1200	~1361	85%
--format brief --budget 600	~963	89%
--format brief --no-text	~423	95%

Five things do the work:

1. Extractive selection, not truncation. Instead of the first N characters, compress.py splits the page into sentences, scores each by how many distinct query terms it covers, keeps the best ones within budget, and re-emits them in document order. Same budget, far higher information density. On the live run the raw prefix spent its tokens on "Turn text into searchable vectors and build semantic search…"; the extract instead kept the definition of an embedding and the trade-offs section.
2. Dropping fields an agent cannot use. score_breakdown, outbound_links, engine, search_rank, from_cache, word_count and snippet are ranking diagnostics, not answers.
3. Compact serialization. Indentation is not free — a newline and each run of indent spaces cost about a token apiece in real tokenizers.
4. Not repeating the field names. JSON re-serializes every key on every result: with eight results, "title", "url" and friends are paid for eight times each, plus their quotes and braces. --format brief is line-oriented and names the fields once, in a header — worth another ~17% over agent at the same budget. Scores are also rounded to 3 decimals; 0.802656 bought nothing over 0.803.
5. Dropping cross-document boilerplate. Cookie banners and newsletter pitches survive extraction because they are real sentences. What gives them away is appearing verbatim on three or more unrelated domains. Honest caveat: this did not fire on the sample above — those eight results share no identical sentences — so it contributes 0% there. It is a safety net for result sets that do overlap, not part of the measured saving.

Budgets are allocated across results with a 1/sqrt(rank) decay, so rank 1 gets meaningfully more room than rank 10 while the tail still gets enough to be worth including. If no sentence matches the query, extraction falls back to a whole-word prefix rather than silently emptying the result.

Token counts are estimates from a dependency-free heuristic (words at ~4 chars per token, punctuation ~0.7, plus layout whitespace), calibrated for English prose and JSON. It is within roughly 10% of GPT-style BPE — accurate enough to budget with, and not worth installing a tokenizer for.

You can also compress a payload you already have, without re-crawling:

.\.venv\Scripts\python.exe -m browser_worker.compress examples\results.json --budget 800 -o compact.json

Rendering with Brave

By default browser-worker never launches a browser — it fetches over HTTP and parses what the server sent. That is blind to anything a page builds with JavaScript.

--render points it at an installed Brave:

# retry only the pages HTTP could not get: 403s and JavaScript-only shells
.\.venv\Scripts\python.exe -m browser_worker "your query" --render auto

# render every page in Brave
.\.venv\Scripts\python.exe -m browser_worker "your query" --render always

Measured on https://quotes.toscrape.com/js/, a page whose content exists only after JavaScript runs:

	Bytes	Detected as shell	Words extracted
HTTP	5806	yes	4
Brave	8925	no	246

Same page, 60× the usable content.

Flag	Default	Effect
--render	never	never, auto, or always
--brave	auto-detect	path to brave.exe; also read from $BROWSER_WORKER_BRAVE
--render-timeout	30.0	seconds before a render is abandoned
--render-concurrency	2	simultaneous tabs — each one is a live page

How it drives Brave, and why that way

Over the DevTools protocol (CDP), not the obvious --headless --dump-dom flag. --dump-dom was tried first and is not dependable on Brave 151: Chromium removed the original headless mode, and under the replacement the flag hangs until killed. CDP is the supported interface and behaves consistently.

That choice pays off twice, because it allows one browser, many tabs. Brave starts once per run and each page is a tab, so browser startup is paid once rather than per URL. A three-page render took 4.2s total, against roughly 2–4s of startup per page for the process-per-page approach.

Three properties worth stating plainly:

• Your real Brave profile is never touched. Every run uses a throwaway --user-data-dir. Your history, cookies, logins, extensions and open tabs are untouched, and an already-running Brave is unaffected — you can browse while it works.
• robots.txt still applies. A real browser is still a bot here; gating HTTP but not rendering would defeat the point of honouring it at all.
• The browser is always shut down. The renderer is owned by the pipeline and closed in a finally, verified by a test. An earlier draft let the fetcher create it lazily with nobody responsible for closing it, which left orphaned Brave processes behind after the run exited.

Rendered pages go through the same cache as HTTP ones, so a re-run does not pay for the browser twice.

Limits

• It is slow: seconds per page against milliseconds for HTTP. That is why the default is never and why auto exists.
• It does not defeat every block. Medium, for instance, still times out under Brave — bot detection sees more than the user agent.
• websockets is required for --render. The import is deferred, so an HTTP-only install never touches it.

How each stage works

Search — browser_worker/search/. DuckDuckGo's server-rendered HTML endpoint, no key needed. Results come wrapped in a /l/?uddg= redirector, which is unwrapped back to the real target. Every backend implements the SearchEngine protocol (name, search(query, limit)), so adding Brave or SearXNG means one new file and one line in the ENGINES registry. When multiple engines run, their result lists are interleaved rather than concatenated, so engine 2's best hit outranks engine 1's tenth.

Fetch — browser_worker/fetch.py. httpx.AsyncClient with a real browser User-Agent, a bounded semaphore (default 5), retry with exponential backoff plus jitter on 408/425/429/5xx (4xx is not retried — it will not change), and a DomainRateLimiter that enforces a minimum gap between requests to the same registrable domain so a run never hammers one host. robots.txt is fetched once per origin, memoized, and fails open — an unreachable or malformed robots.txt is not treated as a blanket disallow.

Extract — browser_worker/extract.py. selectolax. Title from og:title → <h1> → <title>. Publish date from meta tags → <time datetime> → JSON-LD datePublished, normalized to YYYY-MM-DD. Main text by stripping structural junk (nav, footer, aside, script, ad/cookie/newsletter classes) and then picking the densest surviving content block. Outbound links are collected before stripping, absolutized, normalized and filtered to off-site only.

Dedupe — browser_worker/dedupe.py. Two stages, because they catch different things. URL normalization catches the same page reached through different tracking links. Body similarity catches syndicated copies at different URLs: a 64-bit Charikar simhash over 3-token shingles is the cheap pre-filter, and difflib.SequenceMatcher only runs on pairs whose Hamming distance is already ≤ 12, so the expensive comparison is rare.

Rank — browser_worker/rank.py. One file, pure functions, no I/O. Five components, each in [0, 1], combined as a fixed weighted sum:

Signal	Weight	Meaning
coverage	0.40	fraction of query terms present in title+body, plus a small density bonus. Breadth beats repetition on purpose
domain	0.20	hand-tiered quality: arxiv/nature/nih = 1.0, .edu/.gov = 0.9, wikipedia/github = 0.8, unknown = 0.5, spam-shaped = 0.35, content farms = 0.15
recency	0.15	exponential decay, one-year half-life. Unknown date scores a neutral 0.5, not zero
length	0.15	logarithmic, saturating at 1200 words; stubs under 120 words are penalized
title	0.10	query-term coverage of the title alone

Every result carries its score_breakdown in the output, so any ranking decision can be explained by reading five numbers. Editing the weights is a one-line change.

Cache — browser_worker/cache.py. SQLite keyed by normalized URL, so ?utm_source= variants share one entry. The second run of the same query above took 2.5s against 12.9s cold.

Search result lists are cached too, not just page bodies. This was added after repeated identical queries got the run rate-limited into returning zero results: without it, re-asking the same question re-hits the engine every time. Empty result lists are deliberately never cached, so a rate-limited response cannot poison the cache and make the outage permanent.

Tests

.\.venv\Scripts\python.exe -m pytest

284 passed, 0 failed (measured, 2.06s). No test touches the live network — every HTTP interaction goes through httpx.MockTransport. Coverage by area:

File	Tests	What is verified
test_compress.py	66	budget honoured, off-topic sentences dropped, document order preserved, prefix fallback, agent format smaller than indented
test_rank.py	38	each signal in isolation, weights sum to 1, total equals the weighted sum, determinism, good-beats-spam ordering
test_urls.py	36	tracking-param stripping, port/fragment/slash handling, idempotence, rejection of non-http input
test_extract.py	22	title/date fallback chains, junk stripping, link filtering
test_fetch.py	19	retry/no-retry by status, robots enforcement, rate-limit timing, concurrency ceiling, cache short-circuit
test_pipeline.py	19	full run over a mocked network: ranking order, dedupe of a mirrored page, cache hits, all output formats
test_dedupe.py	18	simhash stability, near-duplicate detection, threshold behaviour, kept+dropped partition the input
test_search.py	12	redirector unwrapping, result parsing, endpoint failover
test_search_cache.py	12	second run makes no search request, empty results never cached, limit respected on a hit
test_browser.py	26	render-mode routing, robots gates rendering, failed render falls back, pipeline always closes the browser
test_cache.py	16	TTL expiry, normalized keys, persistence, cache path independent of the working directory

Assumptions

Decisions made without being able to ask. Each is a one-line change if wrong.

1. DuckDuckGo's HTML endpoint is the search layer. It needs no API key, which was the binding constraint. It is also unofficial and rate-limits aggressively under load. lite.duckduckgo.com is wired as an automatic fallback. The engine interface exists precisely so this can be swapped without touching the pipeline.
2. robots.txt fails open. If robots.txt is unreachable or unparseable, the fetch proceeds. Failing closed would make any flaky host look like a blanket disallow. An explicit Disallow is always honoured.
3. Domain quality is a hand-written tier list, not a reputation API. It is short, visible at the top of rank.py, and easy to extend. Unknown domains score a neutral 0.5 rather than being punished for being unknown.
4. Unknown publish dates score 0.5, not 0. Most pages do not declare a date; scoring them zero would systematically bury undated documentation.
5. Non-HTML content types are skipped, PDFs included. Adding PDF extraction means one more dependency and one branch in _finish(); it was out of scope for tonight.
6. registrable_domain() uses a small hardcoded list of two-level public suffixes (co.uk, com.au, …) instead of taking a tldextract dependency. This is imperfect for exotic suffixes; it is only used for rate-limiting and the domain signal, where being slightly wrong is cheap.
7. The result text field is truncated to 2000 characters in the output payload. Full text stays in the cache. This keeps results.json readable; raise text_chars in Document.to_dict() if you need the whole body.
8. Token counts are estimated, not tokenized. The heuristic is within ~10% of GPT-style BPE, which is enough to make a budgeting decision. Adding a tokenizer dependency to count tokens you only use to pick a cutoff is not worth the install. If you need exact counts for billing, count downstream.
9. --format agent implies --lean. Anything consuming the compact form wants answers, not ranking diagnostics. Pass --format json --budget N if you want a budget while keeping the full record.
10. JavaScript-rendered pages return whatever the server sends. There is no headless browser, so SPA shells extract as near-empty and get dropped by the empty-text filter. Live run: 2 of 10 hits failed this way (Medium returned 403 to a non-browser client), which the stats report honestly as fetch_failed.

Known limits

• Medium and other Cloudflare-fronted hosts return 403 to any non-browser client. Counted as a fetch failure and reported, not hidden.
• Ranking is lexical, not semantic. A page that answers the question in different words scores low. Fixing that means embeddings, which is Project 2's job.
• simhash comparison is O(n²) over kept documents. Fine at n ≤ 100; it would need banding beyond that.