BPBench

BPBench measures how well LLMs compress fresh, provably human-written text published after each model's training cutoff — exact bits-per-byte (BPB). A pre-trained LLM is a compressor: the better it models language, the fewer bits it needs to encode text it has never seen. Nothing memorized, nothing gamed.

Leaderboard and writeup: https://research.trelis.com/bpbench

This repo contains everything needed to replicate: the public slice builders, the exact-BPB scorer, all three sampling estimators, the audit tooling, and our estimator outputs. (Our third test slice is private/held-out to catch future benchmark gaming — the pipeline runs on the two public slices out of the box; add your own third via EXTRA_SLICE=<name>.)

Why BPB, not perplexity

Perplexity is per-token, so it's confounded by tokenizer choice. Bits-per-byte normalizes by UTF-8 byte length, making models with different tokenizers comparable:

BPB = (sum of token cross-entropies in bits) / (UTF-8 bytes of the raw text)

Count bytes from the raw text, never from decoded token strings (SentencePiece markers inflate them). Precedent: Compression Represents Intelligence Linearly (Huang et al., 2024).

BPB scores the deployed checkpoint; post-training tends to raise it. Read tiers, not fine rankings.

Layout

data/       congressional.py · hansard.py · make_manifest.py   (public slice builders)
bpb/        score.py          — exact BPB via echo+logprobs (open weights / logprob APIs)
estimator/  sample_open.py    — completions + plain-chat sampled estimates (n=32, 3 pos/doc)
            sample_echo.py    — echo-chat estimate (explicit-quote protocol, min-16 rule)
            sample_closed.py  — chat estimate via OpenRouter w/ reasoning telemetry + cost cap
            leak_audit.py     — reasoning-rate + in-content leak audit per model/channel
            hf_score.py       — transformers teacher-forcing scorer (BOS-explicit; for models
                                vLLM mis-serves)
            tinker_inkling.py / tinker_inkling_estimate.py — Inkling via Tinker (see PROVIDERS.md)
analysis/   plot_channels.py · plot_echo_scatter.py · plot_outliers.py
results/    estimator/ — our sampled-estimate outputs per model (comp/chat/echo + CIs)
PROVIDERS.md — the full method + QA rules (adherence bands, floor rule, telemetry, gotchas)

The estimators, in one paragraph each

• Completions: resample the next token (n=32) on the raw string, no chat template; the match rate estimates the probability. Tracks true BPB on frontier models. Hard limit: a 6-bit floor (an unseen word can only be scored as ~1/64), so the hard-word tail is invisible and weak models read better than they are.
• Plain chat: text in the user turn, ask for the next word. Doesn't work: the first assistant token carries reply-starting probability mass (+~2.7 bits/word measured token-exactly), and the bias is model-dependent (an RLHF fingerprint).
• Echo chat: append Begin exactly with: "<last 8 words>" and extract the word after the echo. Wording matters: quoting the exact string gets 92–100% compliance vs ~2% without. Restores the conditional (verified in logprobs); bias becomes near-uniform but the method remains noisy — we did not consider it reliable enough to place closed models.

Reasoning must be off AND verified (API telemetry); echo adherence measured per model; a position counts only if ≥16/32 samples adhere. Full rules in PROVIDERS.md.

Reproducing

uv sync

# build the public slices
uv run data/congressional.py && uv run data/hansard.py && uv run data/make_manifest.py

# exact BPB (needs FIREWORKS_API_KEY in .env)
uv run bpb/score.py

# sampled estimators (FIREWORKS_API_KEY / OPENROUTER_API_KEY)
uv run estimator/sample_open.py
uv run estimator/sample_echo.py
uv run estimator/sample_closed.py

API keys load from a local .env; none are committed.