reward-hacking-in-the-wild

A raw corpus collector for in-the-wild, user-reported anecdotes of AI-model misbehavior — reward hacking, test tampering, sycophancy, overeagerness, and related failure modes (e.g. "the agent deleted my tests and declared victory"). It pulls from public sources under fully ToS-compliant access, normalizes everything into one shared JSONL envelope, and optionally classifies each item with an LLM.

Classification into categories happens in-pipeline, but the final dataset schema and any integration with reward-hacking-evals are deliberately deferred until the raw data has been examined — "raw corpus first". See the full design rationale in docs/superpowers/specs/2026-07-14-in-the-wild-scraper-design.md.

Sources and compliance posture

Seven sources total: five automated, one manual-only, one stubbed pending access.

Source	Status	Why
Hacker News	automated	Algolia HN Search API is official, public, and requires no auth.
GitHub issues	automated	Uses the authenticated gh CLI against GitHub's own REST/search API — sanctioned access.
LessWrong / Alignment Forum	automated	Public site-search API at /api/search (Algolia-compatible; LW migrated off Algolia proper and its GraphQL endpoint ignores query terms, so this is the only way to keyword-search posts/comments).
AI Incident Database (AIID)	automated	Official weekly JSON snapshot download — no scraping involved.
X (Twitter)	automated, explicit-run only; collected but not published	Official X API v2 recent-search endpoint — see "X source" below for cost/scope caveats. X's developer terms expect posts to be embedded rather than their text rehosted, so X posts are excluded from the published subset (see "Published subset").
Forums (Cursor/OpenAI Discourse)	manual only	Discourse ToS forbids automated access and OpenAI's Terms of Use prohibit automated extraction. Covered by hand-curated ingest-manual (ordinary browser use of URLs the user already selected).
Reddit	stubbed, not enabled	Unauthenticated .json access was shut off in May 2026 and Reddit's terms prohibit unauthorized scraping. The reddit source exists in code but is excluded from scrape all until approved OAuth API access (free non-commercial tier or Reddit for Researchers) is obtained.

Published subset

Collection and publication are separate. Everything scraped lands in data/raw/, but the site and its committed artifacts render only the published subset, defined once in rhw.cases (EXCLUDED_SOURCES, MIN_CONFIDENCE): confidence >= 0.9, excluding AIID and X. Two sources are collected but never republished:

• AIID is CC BY-SA, so republishing it would carry a share-alike obligation over that portion of the dataset.
• X posts are excluded because X's developer terms expect posts to be embedded rather than their text reproduced elsewhere. Linking to a post and quoting a line with attribution are fine; rehosting 2,000-character bodies in a downloadable corpus is not.

build_search_index.py prunes site/public/cases/NN.json files left behind when the subset shrinks — an unreferenced shard stays publicly fetchable, so excluded text would otherwise keep being served.

Attribution note: AI Incident Database content is licensed CC BY-SA. Any published dataset that includes AIID-derived records must attribute AIID and share-alike that portion. Collection for research is the current scope; republishing raises copyright/GDPR questions to revisit at dataset-release time.

Licensing

Three different things live in this repository, with three different owners.

Code — MIT (see LICENSE). The scraper, classifier, severity rater, review tooling, build scripts, and the website.

Annotations — CC BY 4.0. The category labels, the 14-item taxonomy, the severity ratings, and the rationales are original work; reuse them with attribution.

Collected reports — not licensed by this project. The scraped text belongs to the people who wrote it. It is reproduced here as excerpts (2,000 characters maximum) for research and commentary, and every record carries the URL of its source. This project claims no ownership of it and grants no rights over it. If you wrote something included here and want it removed, open an issue and it will be dropped from the published subset.

Two sources are deliberately collected but never republished — see "Published subset" above for why: AIID (CC BY-SA, share-alike) and X (its developer terms expect posts to be embedded rather than their text rehosted).

Nothing here is legal advice, and the boundary between research quotation and redistribution is not perfectly settled. The posture is deliberately conservative: excerpts, attribution, links back, and removal on request.

Setup

uv sync
gh auth login          # required for the github source (uses `gh api search`)
export OPENROUTER_API_KEY=...   # required only for `rhw classify` and `rhw severity`

rhw classify and rhw severity will refuse to run (print an error and exit non-zero) without OPENROUTER_API_KEY set in the environment.

Scraped text flows directly into the classifier prompt, so labels are LLM output shaped by potentially adversarial or misleading source posts — treat them as a research signal to review, not as ground truth.

Commands

All commands run through the rhw console script (uv run rhw <command>) and can be run from any working directory — config paths are anchored to the repo root, not the current directory.

scrape

Collect new items from one source or all automated sources. Dedup is per-source and incremental — re-running only fetches items not already seen.

uv run rhw scrape hn
uv run rhw scrape github
uv run rhw scrape lesswrong
uv run rhw scrape aiid
uv run rhw scrape all      # runs hn, github, lesswrong, aiid (reddit and x excluded)

X source (explicit-run only — costs real money)

x uses the official X API v2 recent-search endpoint (GET /2/tweets/search/recent), which only covers the last 7 days — it is not a historical-collection source. Access is pay-per-use, billed per tweet read (~$0.005/tweet), so it is excluded from scrape all and must be run explicitly:

export X_BEARER_TOKEN=...   # required; put in .env, never commit or print it
uv run rhw scrape x

Spend is bounded by config/config.yaml's x: block — most importantly max_total_tweets (default 5000, ~$25 at $0.005/read), a hard global cap across the whole run that stops collection (with a stderr warning) the moment it's crossed, regardless of how many search terms remain. Also see max_pages_per_term (pages of up to 100 tweets per phrase) to keep any one term from consuming the whole budget.

ingest-manual

Fetch and normalize a hand-curated list of URLs (e.g. forum threads, X posts) already selected by browsing. One URL per line, optional trailing note.

uv run rhw ingest-manual urls.txt

classify

Label unlabeled collected items via OpenRouter. Already-labeled items (keyed by source, id) are skipped unless --relabel is passed.

uv run rhw classify
uv run rhw classify --relabel

severity

Rate real-world harm for the high-confidence subset (non-AIID, relevant: true, confidence ≥ 0.9) on a four-level ordinal scale — negligible, minor, significant, severe. A permanent second stage after classify: already-rated items (keyed by source, id) are skipped unless --rerate is passed. Requires OPENROUTER_API_KEY, same as classify. Ratings land in data/labels/severity.jsonl; malformed model responses degrade to severity: null with a PARSE_ERROR: rationale. The distribution surfaces in rhw stats (by_severity) and on the site's "How bad were they" section.

uv run rhw severity
uv run rhw severity --rerate

stats

Print counts by source, category, relevance, agent, and severity as JSON.

uv run rhw stats

review-sample / review / review-report (label validation)

Human-validate the classifier labels (spec: docs/superpowers/specs/2026-07-19-label-validation-design.md):

uv run rhw review-sample            # stratified ~138-item sample (fixed seed)
uv run rhw review                   # blind review app at http://127.0.0.1:8321
uv run rhw review-report            # writes docs/validation/2026-07-label-validation.md

review-sample draws ≤7 relevant items per category (rarest-category bucketing) plus 40 recall probes from the non-AIID irrelevant pool (20 uniform, 20 with confidence < 0.7). review shows each item blind — the classifier's verdict appears only after you commit yours (y/n, number keys for categories, Enter to commit, u to undo); verdicts append to data/gold/gold.jsonl, the one committed path under data/, holding only (source, id) verdicts, never scraped text. review-report computes relevance and per-category precision with Wilson 95% intervals, a recall estimate, and a disagreement table.

Website (site/)

Live at rewardhacking.org — a Next.js site on Vercel with two routes.

The landing page at / presents the published subset (see "Published subset" above) in four sections: the headline incident count, in-browser corpus search, a category breakdown chart, and the severity distribution ("How bad were they"). /writeup renders the draft essay from site/src/content/writeup.md, a manual copy of the Obsidian draft, with a contents rail, two storyboard strips, and the trend charts.

The site is fully static — everything renders from committed artifacts, because Vercel builds run npm, not Python:

uv run python build_site_data.py     # stats.json — after classify + severity
uv run python build_search_index.py  # search artifacts — after classify

Search runs entirely in the browser over three artifact tiers, each fetched only when needed: search-meta.json (metadata and snippets, on page load), search-postings.json (an inverted index over full case text, on the first keystroke), and cases/NN.json shards (full text, when a case is opened). There is no search server.

Two redirects live in next.config.mjs rather than dashboard state: /search (the page search was folded onto the landing page from) and www, since the apex is canonical.

Deploys are CLI-driven, not Git-triggered — pushing to GitHub does not publish. Note the Vercel project's Root Directory is ., which works only because the deploy runs from inside site/:

cd site && vercel --prod

Local dev: cd site && npm install && npm run dev. Checks: npm test (matching and ranking logic), npm run smoke (boots the built site and asserts each route serves).

Export scripts (site/scripts/)

The essay is crossposted elsewhere, so the figures and a self-contained markdown file are generated rather than hand-assembled. All three drive a local Chrome via puppeteer-core:

node scripts/export-figures.mjs  # storyboards + charts -> export/figures/*.png
node scripts/export-md.mjs       # crosspost .md with the figures inlined as base64
node scripts/export-og.mjs       # public/og.png, the social preview card

export-figures.mjs screenshots the running /writeup page, so the dev server must be up first. Re-run export-og.mjs when the corpus total changes — the headline number is baked into the image.

build_search_index.py

Generate the static artifacts behind the landing page's corpus search — site/public/search-meta.json (metadata and snippets, fetched when search is first used), site/public/search-postings.json (an inverted index over the full case text), and 32 full-text shards under site/public/cases/ (fetched when a case is opened). Search and "I'm feeling lucky" run entirely in the browser over the same high-confidence subset as the rest of the landing page, so no server or search service is involved. Re-run after classify.

uv run python build_search_index.py

Data layout

• data/raw/<source>.jsonl — append-only raw envelope records, one per source.
• data/state/ — per-source seen-ID files, making scrape reruns incremental.
• data/labels/labels.jsonl — one classification label per collected item.
• data/labels/severity.jsonl — one harm rating per high-confidence case (see rhw severity).
• data/relevant_cases.jsonl — every relevant case's label joined with its record fields (url, created_at, text, …); the input all three build scripts read, each applying the high-confidence filter on top.
• data/review/sample.jsonl — the stratified review sample (gitignored).
• data/gold/gold.jsonl — human review verdicts; the one committed path under data/.
• site/public/search-*.json, site/public/cases/ — generated search artifacts (committed; Vercel builds run npm, not Python).
• site/src/data/stats.json — headline counts, category totals, monthly and severity rollups; the landing page renders from it.
• site/export/ — crosspost markdown and figure PNGs (gitignored; regenerate with the export scripts).

data/ is gitignored except data/gold/; nothing else under it is committed.

Schemas

Record envelope (raw corpus)

Every scraper and ingest-manual emits records in one shared shape:

{
  "source": "hn | github | lesswrong | aiid | manual | reddit",
  "id": "source-native unique id",
  "url": "...",
  "created_at": "ISO 8601",
  "author": "username or null",
  "title": "or null",
  "text": "plain text body",
  "matched_terms": ["..."],
  "raw": { "full original API payload" }
}

The raw corpus is never mutated by later stages. (source, id) is the dedup key.

Label (classification output)

One JSON object per classified item in data/labels/labels.jsonl, keyed by (source, id):

{
  "source": "...",
  "id": "...",
  "relevant": true,
  "categories": ["reward_hacking", "..."],
  "agent": "tool name or null",
  "model": "underlying model or null",
  "confidence": 0.0,
  "rationale": "one line",
  "classifier_model": "..."
}

relevant (bool) means the item is an actual first- or second-hand account of a model or agent misbehaving — not abstract discussion, news commentary, or research meta-talk. categories is a multi-label subset of the 14-item taxonomy below (see CATEGORIES in src/rhw/classify.py for the authoritative list):

• reward_hacking — gaming a check/objective (hardcoding expected outputs, faking output, editing the scorer)
• test_tampering — breaking, changing, or deleting tests/logic to conceal non-functional features
• metric_spoofing — spoofing metrics/benchmarks to make a solution look better than it is
• p_hacking — manipulating analyses/statistics until results look significant
• sycophancy — agreeing with or flattering the user against truth/correctness
• overeagerness — doing far more than asked, taking decisive action on vague instructions, ignoring scope
• credential_misuse — misappropriating credentials, keys, or tokens
• unauthorized_access — hacking through systems or interfaces it should not access
• excessive_exploration — over-exploring the workspace/filesystem in a surprising way
• destructive_actions — destructively deleting production code or data
• hidden_backdoors — inserting hidden vulnerabilities or backdoors without disclosure
• unauthorized_communication — reaching out to humans (email, messages, posts) when not instructed
• self_modification — attempting to change its own configuration, permissions, or prompts
• other_misalignment — a concrete misbehavior account that fits none of the above

If the classifier's response is malformed or unparseable, the label degrades gracefully to relevant: null, empty categories, and a rationale prefixed PARSE_ERROR: rather than failing the run.

Severity (harm rating)

One JSON object per rated case in data/labels/severity.jsonl, keyed by (source, id):

{
  "source": "...",
  "id": "...",
  "severity": "negligible | minor | significant | severe",
  "confidence": 0.0,
  "rationale": "one line",
  "rater_model": "..."
}

Severity judges actual damage caused (or nearly caused), not how deceptive the behavior was. Like the labels, ratings are LLM output — a research signal to review, not ground truth.

Configuration

• config/config.yaml — data directory, request delay/retry knobs, GitHub repo list, LessWrong search endpoint/indexes, AIID snapshot URL, OpenRouter model and endpoint.
• config/terms.yaml — shared keyword families (failure phrases, agent phrases, misbehavior phrases) used by hn, github, and lesswrong. aiid and manual don't use search terms — AIID ingests everything and lets the classifier decide, manual takes explicit URLs.

Testing

uv run pytest

Unit tests run against recorded API fixtures — no live network calls in the suite.