v0.11.10 — cost and crawl budget

Nothing a visitor will notice. This release cuts what the site costs to run and stops spending Google's attention on pages that were never worth indexing.

The extractor was 84% of the LLM bill (#564)

extractor is the only pipeline role handed {sentences} — the raw transcript. Every other role works on what it already produced ({events}, {markdown}, {overview}), one to two orders of magnitude smaller. So one role out of seven accounted for 175M prompt tokens in 30 days, ~US$174/month.

The 2026-06 model bake-off that put every role on deepseek-v4-pro judged tag vocabulary and ticker linking — the writer's and ticker_extractor's jobs, on {events}. The role burning the tokens was never the one that decision was about.

Pinned to deepseek-v4-flash-0731, measured across four episodes (537 / 800 / 1102 / 2732 sentences):

	v4-pro	flash-0731
sentence coverage	100%	100%
out-of-range index ranges	0	0
off-vocabulary segment_type	0	0
chapters after consolidation	4 / 8 / 12	4 / 8 / same
cost per episode	—	93–98% lower

It emits more raw events, and that turns out not to matter: consolidate_chapters scales chapter count by episode duration, capped at 12, so how finely the extractor split is normalised downstream. That was the open question, and answering it is why this is a safe swap rather than just a cheap one. The other six roles stay on v4-pro.

Two candidates were rejected on correctness rather than price — qwen3.7-flash left 56 sentences uncovered, ling-3.0-flash cut 800 sentences into 100 events.

The thin topic pages (#560)

/topics/:tag renders 97 characters of its own — nav chrome plus one sentence identical across all ~166 pages — wrapped around a list of links to episode summaries. Those pages now carry X-Robots-Tag: noindex and are gone from the sitemap, which drops from 657 URLs to 488.

They still work; they just stop being something Google evaluates. /topics itself and the 82 sector pages are deliberately kept: the sector pages carry a hand-written thesis and per-constituent descriptions, and /topics hosts the discussion-heat backtest. Those are curation, not filler.

Spotify backfill (#561 #562 #563 #565)

The episode lookup could not see past the newest 100 episodes, ran against the wrong store, paged once per episode instead of once per show (~27,000 requests down to ~240), and could not tell a 429 from a genuine miss. All four fixed; 429 now honours Retry-After and says so.

Verified on staging

$ curl -s https://staging.tinboker.com/sitemap.xml | grep -c '<url>'
488                      # was 657; /topics/:tag went 166 → 0
sector 80 · topics 1 · episode 291 · stock 101 · podcaster 10
$ staging-api /health    200, 0.94s warm

The per-route noindex cannot be verified anywhere but production — every non-prod host already returns the header for a different reason (v0.11.9 stamps dev/staging wholesale). Same for the extractor pin: it lives in a systemd unit only the production deploy installs.

After this deploys

• Watch one ingest slot (02/08/14/20:10 TW) and confirm the episodes have real chapters, not placeholder summaries. That is the first real exercise of the new extractor model.
• /topics/ai should return noindex on tinboker.com; /topics should not.
• Warm /sitemap.xml once — it regenerates cold after a container restart (45s last time, ~1.4s warm).

Full changelog: v0.11.9...v0.11.10