Slopindex compares the recurring language patterns of AI models. It runs a fixed prompt suite, stores each generation by content identity, and builds model and document-level Jaccard comparisons.
Slopindex is an independent Bun and Effect v4 implementation inspired by Sam Paech's slop-forensics. That project supplies the benchmark prompts, reference outputs, derived profiles, and feature lexicons under corpora/. The data is pinned to upstream commit c313f042620f027d49101da3256bd306b628071a and remains under its MIT license. This repository does not copy the upstream Python implementation. See THIRD_PARTY_NOTICES.md for the full attribution.
Each model profile contains the presence of its most common words, bigrams, and trigrams. Pairwise similarity is the Jaccard score:
shared features / combined features
The public model score blends every prompt collection available to a pair. Profiles from the same model never compare with each other, including runs at different temperatures. Head-to-head pages use the same feature-presence method for each aligned pair of outputs.
Profiles are not all based on the same number or kind of outputs. At launch, most newly generated frontier profiles contain 99 or 100 essay outputs. The imported reference corpus contains 734 to 2,000 essay outputs per model and 74 to 1,999 varied-prompt outputs, but its coverage is also uneven. A pair only blends prompt collections that both models have, so comparisons involving a frontier model are usually essay-only today.
Treat an n near 100 as exploratory. It can expose strong recurring language, but small score differences and exact leaderboard positions are not conclusive. A stronger profile should first extend the same essay configuration to about 500 outputs and add a varied-prompt run. If the result still changes materially as n grows, extend it toward 1,000 outputs.
Generating new outputs costs money. Every uncached request runs through Vercel AI Gateway. The CLI always estimates the current cost before execution and requires an explicit cost ceiling. Exact requests are cached by identity, so extending an unchanged model configuration reuses its existing outputs and bills only the missing requests.
Slopindex does not store an AI Gateway key in the canonical GitHub repository, and its paid workflow is disabled there. To run the suite with GitHub Actions, fork the repository, add VERCEL_AI_GATEWAY_API_KEY under Settings → Secrets and variables → Actions in your fork, and run the Add model profile workflow from the fork. Leave Execute paid run off first to get a live estimate without sending model requests. Turn it on only after you review the estimate and cost ceiling, then open a pull request with the generated data.
You can also run the same flow locally with your own Vercel AI Gateway account:
bun install
cp .env.example .env
bun run slopindex cost \
--model thinkingmachines/inkling \
--profile essays \
--samples 500
bun run slopindex run \
--model thinkingmachines/inkling \
--profile essays \
--samples 500 \
--max-cost-usd 10 \
--executeSet VERCEL_AI_GATEWAY_API_KEY in .env before a paid run. Replace the example cost ceiling with one that you reviewed and accept. If the estimate is acceptable, run with --execute, run bun run check, and open a pull request with the new cache/generations, runs, and profiles files. Include the model ID, prompt collection, temperature, output count, estimate, and actual cost in the pull request.
corpora/ pinned source prompts, outputs, and features
cache/generations/<prefix>/ one content-addressed JSON file per generation
runs/<run-id>/ manifest, generations, analysis, and comparison
profiles/index.json small generated registry index
profiles/models/.../profile.json one generated model profile
profiles/models/.../documents/ one JSON file per prompt and output
profiles/comparisons/ one comparison list per prompt collection
The source corpus stays in its upstream layout for provenance. Routine model PRs change granular cache, run, profile, and document files. They do not rewrite one large registry artifact.
bun run check
bun run playground
# http://127.0.0.1:4173Build the deployable TanStack Start server with:
bun run playground:buildThe playground uses full-document server rendering. Model data loads in a parent route, while each prompt loads in a child route with adjacent pages preloaded for navigation. The Bun workspace also contains focused packages for the domain model, corpus access, Vercel AI Gateway client, generation cache, run store, profile registry, runner, and Effect CLI. Runtime Effect packages are pinned to 4.0.0-beta.99.
Slopindex is MIT licensed. Imported benchmark data keeps the upstream copyright and license recorded in corpora/UPSTREAM_LICENSE and THIRD_PARTY_NOTICES.md.