HOLMES — Hypothesis-driven Optimization via LLM-guided Model Exploration and Search
HOLMES benchmarks hyperparameter optimization of an implicit-feedback ALS recommender on the
Amazon Reviews 2023 dataset — any category (Books is the default; preprocess one or --all of them
side by side) — comparing four strategies on the same fit budget:
- grid — exhaustive grid search
- random — random search (i.i.d. samples over the same space)
- bayes — Bayesian optimization (Optuna TPE)
- holmes — an agentic loop where an LLM reasons over a diagnostic battery, forms a falsifiable hypothesis each round, and proposes the next hyperparameters as a test of it
The shared ALSRecommender, diagnostic battery, and evaluation harness mean every strategy
optimizes the identical objective (held-out NDCG@10) on the identical splits.
There are two ALS algorithms in the literature, and we deliberately use the implicit variant despite the data being rating-shaped:
- Zhou et al. 2008 (Explicit ALS) — for rating prediction. Loss is
Σ_observed (r_ui − x_u·y_i)² + λ(...). Only observed entries enter the loss; unobserveds are "missing." This is the Netflix-Prize-style algorithm. - Hu et al. 2008 (Implicit ALS) — for engagement signals. Loss is
Σ_all c_ui (p_ui − x_u·y_i)² + λ(...). All user-item pairs contribute, with observed = high-confidence positive and unobserved = low-confidence zero. This is whatimplicit.als.AlternatingLeastSquaresimplements.
For Amazon reviews (Books or any other category), implicit ALS is the right choice, despite the data being rating-shaped:
- Most users haven't reviewed most books — the unobserveds are missing not at random (didn't encounter), not "disliked." Explicit ALS would silently treat them as missing and only train on observed entries, throwing away the strongest signal in the dataset.
- The task is top-K ranking (NDCG@10 on a held-out next item), not rating prediction. Ranking is exactly what implicit ALS optimizes.
- Hu et al.'s
c_ui = 1 + α·r_uiwas designed to consume a continuous engagement signal — they used TV watch-time; we use the review's star rating (stored as the matrix value rather than a binary 1, so a 5-star carries ~25% more confidence than a 4-star). Using the rating is the framework's intended use, not a hack.
If the goal were to predict the rating a user would give a book, Zhou-style explicit ALS (or SVD++) would be right. But for "what should we recommend?", implicit ALS with rating-weighted confidence is the principled choice.
The framework can ingest any Amazon Reviews 2023 category (preprocess --category <name>), but the
benchmark is fixed to a pre-registered subset of 7 — broad enough for the optimizer comparison to
generalize, small enough to run at a sane compute budget. preprocess --all builds exactly these 7.
Selection criteria, fixed before running any optimizer (so the choice can't be biased by results): span the size range — from ~10⁴ to ~10⁷ interactions (small → very large) — across two domains, media (e.g. CDs, Books) vs. physical goods. The result spans ~2.5 orders of magnitude in size and the full density range.
| category | interactions | users | items | density | int/user | domain | scale tier |
|---|---|---|---|---|---|---|---|
| Arts_Crafts_and_Sewing | 25,511 | 3,338 | 3,441 | 0.222% | 7.6 | goods | small |
| Video_Games | 406,724 | 63,413 | 19,022 | 0.034% | 6.4 | media | small–mid |
| Baby_Products | 615,003 | 101,429 | 27,702 | 0.022% | 6.1 | goods | medium |
| CDs_and_Vinyl | 1,019,946 | 101,986 | 74,645 | 0.013% | 10.0 | media | medium |
| Automotive | 3,397,370 | 457,844 | 202,814 | 0.004% | 7.4 | goods | large |
| Books | 5,903,332 | 603,422 | 395,385 | 0.002% | 9.8 | media | large |
| Electronics | 8,260,845 | 1,145,516 | 284,008 | 0.003% | 7.2 | goods | very large |
(Stats are post-preprocessing: after dedup, a 5-core filter on users and items, and the
leave-last-out split. int/user = interactions per user, a repeat-engagement signal that tends to
run higher for media — clearest for CDs and Books.)
Two categories are called out as deliberate exclusions (the tempting tiny and huge ones):
Gift_Cards (123 items — top-K ranking is trivial) and Kindle_Store (a redundant very-large
media set already represented by Electronics at that scale tier). Neither is in AMAZON_CATEGORIES;
either can be rebuilt on demand with preprocess --category <name> (e.g. Gift_Cards for a quick
smoke test).
All reported results are generated on a single instance type, so every strategy scores each
config on an identical objective — the comparability the benchmark enforces by construction (the
shared evaluate_config and TestComparabilityInvariants).
This project uses uv for dependency and environment management.
uv sync# 1. Build the interaction matrix from any Amazon Reviews 2023 category. The first run downloads the
# gzipped reviews (~20GB uncompressed for Books) from the McAuley mirror and caches the columns it
# needs as data/raw_cache/<category>.parquet; later runs reuse that cache and skip the download.
# Polars then streams the dedup, k-core filter, and leave-last-out split into
# data/processed/<category>/ — so each category is a separate, side-by-side dataset.
uv run holmes preprocess # Books (default), full dataset
uv run holmes preprocess --category Electronics # any category by name
uv run holmes preprocess --category Video_Games --max-interactions 2000000 # cap rows for a quick dev matrix
uv run holmes preprocess --all # every category into data/processed/<category>
# Point --data at the category subdirectory you preprocessed; it's required on every command below.
#
# Each run fits ONE seed. For stability across initializations, repeat a run with different
# --seed values (and --search-seed for random/bayes) and aggregate the results yourself.
# `holmes dispatch` runs that whole baseline grid (grid/random/bayes x categories x seeds)
# unattended, one job at a time, skipping any whose result JSON already exists.
# 2a. Grid-search baseline.
uv run holmes grid --data data/processed/Books --seed 0
# 2b. Random-search baseline (--seed is the per-fit seed; --search-seed draws the configs).
# Like grid, it always runs the shared fit budget; only the trajectory seed is configurable.
uv run holmes random --data data/processed/Books --seed 0 --search-seed 0
# 2c. Bayesian-optimization baseline (--seed is the per-fit seed; --sampler-seed is the TPE
# search trajectory). Like grid and random, it always runs the shared fit budget.
uv run holmes bayes --data data/processed/Books --seed 0 --sampler-seed 0
# 2d. The agentic HOLMES loop is driven by an LLM reading holmes/sweep_prompt.txt. Each round runs
# ONE iteration, appending diagnostics to an append-only trajectory log. Iteration 1 is a
# normal iteration: the LLM reads `holmes ranges` (HP bounds, budget, dataset signal) and
# chooses its own starting config — there is no separate seeding step. `holmes sweep` runs
# the whole sandboxed sweep unattended (one `claude -p` session per trial).
uv run holmes ranges --data data/processed/Books # HP bounds, budget, dataset signal
uv run holmes holmes-iter --data data/processed/Books --trajectory results/trajectory.json \
--seed 0 --factors 128 --regularization 0.1 --iterations 22 --alpha 22.0 \
--mechanism "..." --outcome "..." --falsifiers "..."
# The hypothesis above is written BEFORE the fit; `annotate` closes the loop after it, recording
# how the prediction fared. The agent calls this itself each round — it is listed here so a
# trajectory can be read (or repaired) by hand.
uv run holmes annotate --trajectory results/trajectory.json --iteration 1 \
--status validated --interpretation "Gap fell as predicted and ndcg rose; the lever is regularization."
# 3. Score a chosen config on the held-out test split for an unbiased number.
uv run holmes eval --data data/processed/Books --params '{"factors": 96, "regularization": 0.05, "iterations": 30, "alpha": 20.0}' --split testThe HOLMES strategy is not fully scripted — it is run by an LLM reading the trajectory between
iterations. holmes sweep runs the trials unattended, one sandboxed headless claude -p session
per trial (holmes sweep --help for the knobs).
Everything that LLM reasons from is inlined into a single prompt template,
holmes/sweep_prompt.txt — there is no skill to load and no reference file to open, so a trial
depends on nothing but the prompt it is handed:
- the autonomous loop: hypothesize → run one iteration → interpret → repeat;
- the hypothesis discipline (mechanism + outcome + falsifiers, and the four validation states) and the diagnostic metrics each fit reports.
tests/test_sweep_prompt.py locks the prompt's metric vocabulary to the diagnostic battery, so a
renamed metric can't leave the prompt hypothesizing about a key nothing measures.
The prompt forbids recommending a config more than 3% below the best ndcg measured; within that
band the rest of the battery decides. The rule exists because three earlier Books runs recommended
configs they had measured as 5.7–9.5% worse, justified by a better train_test_ndcg_gap.
3% is the seed-noise floor, measured free from the grid baselines: grid fits the same 72 configs at
seeds 0/1/2, so (max - min) / mean across seeds — over each category's top 10 by mean score, the
only band a tie-break can fire in — gives a CV of 0.7–4.8%. 3% covers five of the six categories
and blocks all three failures. 5% is too wide: on four categories the whole top 10 sits within 3.2%
of the best, so it would hand the battery the winner outright. 1% is under the noise everywhere.
That top-10 spread also means "best validation NDCG" is substantially seed luck on those four
categories, for every strategy — report across the three seeds with a spread, not a single number.
uv run pytest # run the test suite
uv run pytest --cov=holmes # with coverage
uv run ruff format . && uv run ruff check .holmes/
config.py # hyperparameter spaces, evaluation settings, ALSParams
data/
preprocess.py # any Amazon Reviews 2023 category -> sparse matrix + leave-last-out splits
dataset.py # on-disk Dataset container
als/model.py # ALSRecommender (implicit wrapper) shared by every strategy
metrics/diagnostics.py # the diagnostic battery
search/
harness.py # fit one config (one seed), compute the diagnostic battery
grid.py # grid search
random_search.py # random search
bayes.py # Optuna Bayesian optimization
holmes.py # run ONE HOLMES iteration, append to trajectory
trials.py # per-trial state machine and sandbox for the `holmes sweep` runner
sweep.py # the `holmes sweep` runner: one sandboxed claude session per trial
sweep_prompt.txt # the whole agentic loop inlined: instructions + hypothesis rules + metrics
dispatch.py # enumerate the baseline (grid/random/bayes) runs; `holmes dispatch` runs pending ones here
cli.py # `holmes preprocess|grid|random|bayes|holmes-iter|ranges|annotate|eval|sweep|dispatch`
tests/ # test suite