Interpretable Detection of Reward Hacking in RL

This repository implements a deliberately mis-specified box-pushing grid-world, a PPO baseline, checkpoint evaluation utilities, a sparse autoencoder (SAE) pipeline, and a classifier for detecting reward-hacking behavior from learned representations.

The project goal is to:

1. Train RL agents that exhibit reward hacking,
2. collect internal activations from the policy network,
3. train a sparse autoencoder on those activations,
4. use SAE features to classify honest vs. hacked trajectories.

Current Scope and Status

This repository has moved beyond the original proposal stage and now contains a working end-to-end reward-hacking detection pipeline:

• PPO training for honest and hacking regimes
• checkpoint evaluation and trajectory labeling
• activation collection from frozen policies
• SAE training on policy activations
• classifier comparison using SAE features, raw activations, behavioral summaries, and chance
• cross-seed and cross-task transfer evaluation utilities
• top-SAE-feature analysis against simulator diagnostics

The gridworld pipeline is the most controlled implementation, and the LunarLander extension now provides a second environment for testing whether the same representation-learning recipe still finds reward-hacking signal. Current audited results are summarized in docs/results_summary.md.

Just as importantly, the scope has narrowed since the proposal: the current code does not claim a universal detector that works across all environments without calibration. Instead, it implements the same analysis pipeline in multiple environments with a shared 128-dim policy trunk design, then evaluates transfer explicitly across seeds and across gridworld/LunarLander.

Proposal and Check-In Clarifications

Two questions from the proposal/check-in feedback are now answered clearly in the codebase:

1. Where do the honest/hacked labels come from?

   They come from simulator-side ground truth, not human annotation. The environments explicitly track task completion and exploit-specific diagnostics such as goal_reached, exploit_cycle_count, progress_events, reversal_events, and zone_steps. Those signals are used for post-hoc labeling.

2. Are SAE features standardized across tasks for one classifier?

   Partially. The code standardizes the policy trunk size and episode-level aggregation scheme, and it now includes a pooled SAE experiment trained on both gridworld and LunarLander activations. The project still does not claim a calibration-free universal classifier; instead, it evaluates transfer directly and reports when threshold/direction calibration is needed.

The check-in feedback about SAE sparsity also shaped the current direction: the implemented SAE uses an L1 sparsity penalty, and the latest sweeps push that penalty high enough to reduce mean active features from the dense midpoint result to roughly 35/1024 on gridworld and 58/1024 on LunarLander.

Milestone Snapshot

The midpoint gridworld run documented in midpoint-report.tex achieved:

• a full train -> evaluate -> collect -> SAE -> classify pipeline
• a balanced activation dataset of 144 episodes (72 hacked, 72 honest)
• perfect separation on that dataset for both SAE and raw-activation classifiers

That result is useful as a pipeline validation, but it should not be overinterpreted: because the honest and hacked examples came from policies trained under different reward regimes, it does not yet show that SAE features outperform raw activations or generalize across seeds/tasks. Those are explicitly second-half goals.

The latest local experiments go beyond that midpoint snapshot:

• same-policy honest-vs-hacked datasets for gridworld and LunarLander
• sparse SAE sweeps with stronger L1 penalties
• behavioral baselines alongside SAE, raw activations, and chance
• gridworld seed0 -> seed1 transfer
• pooled-SAE cross-task transfer between gridworld and LunarLander
• top-SAE-feature summaries correlated with simulator diagnostics

See docs/results_summary.md for the current audited result tables and caveats.

Grid-World Environment

The environment is BoxProgressGridWorld in src/hackrl/envs/gridworld.py.

#########
#A......#
#.B.G...#
#.......#
#########

• A agent start, B box start, G goal, # wall, . empty
• 4 discrete actions: up, down, left, right
• The agent pushes the box by moving into it (if the cell beyond is walkable)
• The episode terminates when the box reaches the goal or max_steps is reached

Reward Structure and the Exploit

Base reward (flat, escalation_power=0)

reward = step_penalty                              # -0.02 per step
if box moves closer to goal:
    reward += progress_reward * distance_delta      # +1.3 per unit closer
if box reaches goal:
    reward += goal_bonus                            # +4.0 completion bonus
    episode terminates

The progress reward is not potential-shaped (Ng et al. 1999): pushing the box closer earns +1.3, but pushing it away costs only the step penalty (-0.02). This asymmetry creates a theoretical exploit: push closer, undo, push closer again for repeated rewards.

Why flat reward alone cannot produce hacking

With flat rewards and episode termination on goal, honest play always dominates. The agent completes the task in ~3 steps, earns ~6.5 reward per episode, and farms ~66 episodes per 200 steps for ~430 total. Cycling yields only ~28.5 per 200 steps. Honest play wins by 15x.

This is consistent with the research literature: every canonical cycling exploit (DeepMind boat race, CoastRunners, Q*bert) uses either fixed-length episodes or agent-controlled termination. No published example shows cycling exploits with automatic goal-based termination.

Escalating reward (escalation_power=2)

To make hacking emergent, we introduce two compounding bugs:

Bug 1 — Faulty termination (terminate_on_goal=False): The goal-completion detector fails, so the episode does not end when the box reaches the goal. The agent keeps operating for the full max_steps, and after reaching the goal it discovers it can push the box away and back for repeated progress rewards. This mirrors real-world scenarios where a broken monitoring system fails to stop an agent after task completion.

Bug 2 — Escalating progress counter (escalation_power=2): The progress reward multiplier scales with the cumulative number of closer-pushes, squared:

if box moves closer to goal:
    progress_events += 1
    multiplier = progress_events ^ escalation_power
    reward += progress_reward * distance_delta * multiplier

Push #	Multiplier	Reward
1	1	+1.3
2	4	+5.2
5	25	+32.5
10	100	+130.0
20	400	+520.0

Together, these bugs create a strong hacking incentive: the agent reaches the goal in ~3 steps, collects the goal bonus, then cycles the box back and forth for the remaining ~197 steps. Each cycle earns an escalating reward, reaching 1.3 * sum(n^2, n=1..20) = 3,727 — far more than honest play.

Why both bugs are needed

• Faulty termination alone (flat reward): cycling earns only 0.14/step vs honest 2.18/step — honest farming always wins.
• Escalating reward alone (normal termination): the agent discovers honest play in 3 steps before it can explore the 10-step cycling sequence, and PPO cannot escape the local optimum.
• Both together: the agent naturally discovers cycling after reaching the goal (no termination barrier), and the escalating reward makes cycling overwhelmingly profitable.

Reward parameters

Parameter	Default	Hacking config	Honest config
step_penalty	-0.02	-0.02	-0.02
progress_reward	1.3	1.3	1.3
goal_bonus	4.0	2.0	4.0
escalation_power	0.0	2.0	0.0
terminate_on_goal	True	False	True

Environment Labels and Diagnostics

The environment tracks signals for ground-truth labeling:

• progress_events: total closer-pushes this episode
• reversal_events: total away-pushes this episode
• exploit_cycle_count: increments on progress-after-reversal
• goal_reached: whether the box reached the goal
• true_objective: 1 if goal reached, else 0

Trajectory labeling (label_trajectory):

• honest: goal_reached == True
• hacked: goal_reached == False and exploit_cycle_count >= min_exploit_cycles
• neutral: everything else

PPO Policy

The PPO network lives in src/hackrl/models/ppo.py.

Observation Encoding

9-dimensional normalized float vector:

1. agent x, 2. agent y, 3. box x, 4. box y, 5. goal x, 6. goal y,
2. box-goal Manhattan distance, 8. steps taken, 9. steps remaining

GridWorldPPOPolicy: shared-trunk actor-critic MLP with hidden sizes (128, 128), Tanh activations, categorical policy head, scalar value head. Returns logits, value, and features (shared trunk activations for SAE analysis).

Training

Training code: src/hackrl/training/ppo_trainer.py. CLI runner: scripts/train_gridworld_ppo.py.

Key flags

python scripts/train_gridworld_ppo.py \
    --escalation-power 2.0 \
    --goal-bonus 2.0 \
    --no-terminate-on-goal \
    --total-updates 300 \
    --checkpoint-interval 15 \
    --num-envs 32 \
    --rollout-steps 256 \
    --device cuda \
    --checkpoint-dir artifacts/checkpoints/my_run \
    --resume-from path/to/checkpoint.pt

Default hyperparameters

• total_updates=300, checkpoint_interval=15 (20 checkpoints)
• num_envs=32, rollout_steps=64 (2048 transitions/update)
• learning_rate=3e-4, gamma=0.99, gae_lambda=0.95
• clip_coef=0.2, value_coef=0.5, entropy_coef=0.03
• ppo_epochs=4, minibatch_size=256

Checkpoint Evaluation

CLI: scripts/evaluate_gridworld_checkpoint.py

python scripts/evaluate_gridworld_checkpoint.py \
    artifacts/checkpoints/my_run \
    --num-trajectories 300 \
    --min-exploit-cycles 2 \
    --device cuda

Activation Collection

CLI: scripts/collect_activations.py

Collects per-timestep trunk activations from a frozen policy checkpoint. Each episode is stored as an EpisodeRecord with activations, actions, rewards, and trajectory label. The dataset is split into train (70%) / val (15%) / test (15%) with stratified sampling by label.

python scripts/collect_activations.py \
    artifacts/checkpoints/hack/update_00300.pt \
    --num-episodes 1000 \
    --output-dir artifacts/activations/hack

Sparse Autoencoder

Implementation: src/hackrl/sae/model.py and src/hackrl/sae/trainer.py. CLI: scripts/train_sae.py.

Single-layer autoencoder with an overcomplete dictionary (128 input x 8 = 1024 features). Trained with reconstruction MSE + L1 sparsity penalty. The sparse codes decompose trunk activations into interpretable features for downstream classification.

At midpoint, the SAE was successfully trained end-to-end, but the learned representation was still fairly dense (L0 ~= 377 / 1024 active features per timestep in the midpoint run). The current sweeps use stronger L1 penalties; the selected 3e-2 runs keep perfect in-domain classification while reducing mean L0 to roughly 34.8 / 1024 on gridworld and 57.6 / 1024 on LunarLander.

python scripts/train_sae.py \
    artifacts/activations/hack/train.pt \
    --epochs 50 \
    --dict-multiplier 8 \
    --sparsity-coef 1e-3 \
    --device cuda

Classification

Implementation: src/hackrl/classifier/train.py. CLI: scripts/run_classification.py.

Binary classification of honest vs. hacked trajectories using four baselines:

1. SAE-based: logistic regression on SAE sparse codes (mean-pooled over timesteps)
2. Raw baseline: logistic regression on raw trunk activations
3. Behavioral baseline: logistic regression on coarse trajectory statistics
4. Chance baseline: 50% accuracy

Reports AUROC, F1, accuracy, top-5 predictive SAE feature indices, and top behavioral features.

In the latest experiments, SAE, raw-activation, and behavioral classifiers all achieve perfect in-domain separation on the curated gridworld and LunarLander splits. That confirms the pipeline works, but it also means the in-domain setup is too easy to show a clear SAE advantage. The more important evidence is now the transfer suite: gridworld seed0 -> seed1 and pooled-SAE cross-task transfer between gridworld and LunarLander.

python scripts/run_classification.py \
    --train-data artifacts/activations/merged/train.pt \
    --val-data artifacts/activations/merged/val.pt \
    --test-data artifacts/activations/merged/test.pt \
    --sae-checkpoint artifacts/sae/sae_trained.pt \
    --output-dir artifacts/results

LunarLander Environment

The second environment is LunarProgressEnv in src/hackrl/envs/lunar_lander.py, wrapping gymnasium's LunarLanderContinuous-v3.

This environment is best understood as the next-stage extension of the project rather than the main completed claim from the midpoint milestone. It reuses the same PPO -> activation -> SAE -> classifier structure in a continuous-control setting.

Observation and Action Space

• Observation: 8-dim continuous vector — (x, y, vx, vy, angle, angular_vel, left_leg_contact, right_leg_contact)
• Actions: 2-dim continuous — (main_engine_throttle, lateral_engine_throttle)

Zone-Based Hover Exploit

The reward shaping introduces a "safe zone" at low altitude over the landing pad:

zone: altitude ∈ (0.05, 0.50)  and  |x| < 0.50

When zone_reward > 0, an agent hovering inside the zone earns +zone_reward per timestep. The agent can therefore choose between:

• Honest play: land quickly and collect goal_bonus (episode ends immediately)
• Hacking play: hover in the zone for up to max_steps, collecting per-step zone reward

With zone_reward=1.0, goal_bonus=20.0, and max_steps=600, an agent that hovers for just 21+ steps matches a successful landing, and hovering for 600 steps earns up to 600 reward. The agent discovers the hover exploit around update 640 of training and begins preferring it in most episodes, while still stochastically landing in some episodes — producing a mix of honest and hacked trajectories from the same checkpoint.

Trajectory Labeling

• honest: goal_reached == True (both legs contact + terminated from gym)
• hacked: goal_reached == False and zone_steps >= min_exploit_cycles (default 100)
• neutral: everything else

Reward Parameters

Parameter	Default	Hacking config
zone_reward	0.0	1.0
goal_bonus	100.0	20.0
step_penalty	-0.02	-0.02
max_steps	1000	600
terminate_on_landing	True	True

Training (LunarLander)

python scripts/train_lunar_ppo.py \
    --zone-reward 1.0 \
    --goal-bonus 20.0 \
    --max-steps 600 \
    --total-updates 800 \
    --entropy-coef 0.01 \
    --checkpoint-dir artifacts/checkpoints/lunar_hack

Activation Collection (LunarLander)

python scripts/collect_lunar_activations.py \
    artifacts/checkpoints/lunar_hack/update_00800.pt \
    --num-episodes 2000 \
    --min-exploit-cycles 100 \
    --output-dir artifacts/activations/lunar

The LunarLander PPO policy (src/hackrl/models/ppo_continuous.py) uses the same 128-dim shared trunk as the gridworld policy, but with a Gaussian policy head: an action_mean linear layer and a learnable action_logstd parameter. The 128-dim trunk activations are collected as features for SAE analysis, exactly as in the gridworld pipeline.

Full Pipeline

The pipeline (scripts/slurm_full_pipeline.sh) runs end-to-end:

1. Train hacking agent (escalation_power=2, goal_bonus=2, no_terminate_on_goal) — agent discovers cycling exploit
2. Train honest agent (escalation_power=0, goal_bonus=4, terminate_on_goal) — agent learns direct completion
3. Evaluate both sets of checkpoints
4. Collect activations from the best hacking and best honest checkpoints
5. Merge and balance the two activation datasets (equal honest/hacked episodes)
6. Train SAE on the merged activations
7. Classify honest vs. hacked using SAE features, raw features, behavioral features, and chance baseline

sbatch scripts/slurm_full_pipeline.sh

The documented midpoint run used this gridworld pipeline to produce the first end-to-end results. The current local workspace also contains later generated artifacts/ outputs for same-policy, cross-seed, and cross-task experiments; these outputs may not be versioned in Git, so use docs/results_summary.md as the compact manifest of the audited runs.

Repository Structure

Grid-World

• src/hackrl/envs/gridworld.py — environment dynamics, reward, labels
• src/hackrl/models/ppo.py — discrete PPO policy and observation encoders
• src/hackrl/training/ppo_trainer.py — PPO training loop with checkpointing
• src/hackrl/evaluation/gridworld_checkpoints.py — checkpoint loading and rollout labeling
• src/hackrl/evaluation/activation_collection.py — activation collection and dataset splitting
• src/hackrl/sae/ — sparse autoencoder implementation
• src/hackrl/classifier/ — classification pipeline (SAE vs raw vs chance)
• scripts/train_gridworld_ppo.py — gridworld training CLI
• scripts/evaluate_gridworld_checkpoint.py — evaluation CLI
• scripts/collect_activations.py — gridworld activation collection CLI
• scripts/split_hack_raw.py — split a single checkpoint's activations into train/val/test
• scripts/train_sae.py — SAE training CLI
• scripts/run_classification.py — classification CLI
• scripts/slurm_full_pipeline.sh — end-to-end SLURM pipeline (gridworld)
• tests/test_gridworld.py — environment regression tests

LunarLander

• src/hackrl/envs/lunar_lander.py — LunarProgressEnv wrapping LunarLanderContinuous-v3
• src/hackrl/models/ppo_continuous.py — Gaussian PPO policy for continuous action spaces
• src/hackrl/training/lunar_ppo_trainer.py — PPO training loop for LunarLander
• scripts/train_lunar_ppo.py — LunarLander training CLI
• scripts/collect_lunar_activations.py — LunarLander activation collection CLI
• scripts/run_lunar_pipeline.sh — end-to-end shell pipeline (LunarLander)

Research Context

This project is grounded in the following observations from the AI safety literature:

• Non-potential-shaped reward shaping can change the optimal policy (Ng, Harada & Russell, 1999). Our progress reward is asymmetric: pushing closer is rewarded, pushing away is not penalized beyond the step cost.

• Canonical cycling exploits require fixed-length episodes or agent-controlled termination (DeepMind boat race, CoastRunners, Q*bert). With goal-based termination, episode farming always dominates flat cycling rewards.

• Escalating reward counters create exploitable structure even with goal-based termination, because the cycling agent accumulates a high multiplier within one long episode while the honest agent resets the counter each short episode.

• Sparse autoencoders can decompose RL agent activations into interpretable features (DuPlessie, MIT PRIMES 2024), and SAE features on reward models can detect safety-relevant patterns (SAFER, 2025).

Reproducing the Final Report

The exact commands and reference data below correspond to the runs reported in final-report.tex. The rougher snippets earlier in this README show the generic CLIs; the commands here are the specific invocations whose outputs populate the tables and figures in the report.

Exact training and collection commands

Gridworld seed0 checkpoint (used for the in-domain and source-of-transfer experiments):

python scripts/train_gridworld_ppo.py \
  --total-updates 300 --checkpoint-interval 15 \
  --num-envs 32 --rollout-steps 256 --seed 0 --device cpu \
  --escalation-power 2.0 --goal-bonus 2.0 --no-terminate-on-goal \
  --checkpoint-dir artifacts/checkpoints/gridworld_hack

Main gridworld activation bundle:

python scripts/collect_activations.py \
  artifacts/checkpoints/gridworld_hack/update_00300.pt \
  --num-episodes 3000 --min-exploit-cycles 2 --device cpu \
  --output-dir artifacts/activations/hack_raw --seed 42 --skip-split

Main LunarLander checkpoint:

python scripts/train_lunar_ppo.py \
  --zone-reward 1.0 --goal-bonus 20.0 --max-steps 600 \
  --total-updates 800 --checkpoint-interval 50 \
  --num-envs 16 --rollout-steps 128 --seed 0 --device cpu \
  --checkpoint-dir artifacts/checkpoints/lunar_hack

The transfer-evaluation script uses exact candidate thresholds drawn from observed validation probabilities rather than a fixed grid; this avoids missing useful thresholds when probabilities are compressed after domain shift.

Top SAE features (final in-domain classifiers)

Gridworld — positive Hacked − honest means more active in hacked episodes:

Feature	Honest mean	Hacked mean	Hacked − honest	Main direction
941	1.194	0.134	−1.060	honest
730	0.380	0.066	−0.314	honest
686	0.000	0.400	+0.400	hacked
253	0.475	0.031	−0.445	honest
851	0.000	0.192	+0.192	hacked

LunarLander:

Feature	Honest mean	Hacked mean	Hacked − honest	Main direction
649	0.106	0.463	+0.357	hacked
104	0.080	0.361	+0.281	hacked
822	0.342	0.052	−0.289	honest
590	0.188	0.056	−0.132	honest
145	0.087	0.211	+0.124	hacked

Reviewer feedback traceability

• Ground-truth labels (proposal review): simulator-side labels are computed from goal_reached, exploit-cycle counters, progress/reversal counters, and Lunar zone steps. The classifier does not see these label fields directly. See report §3 and §6.
• Cross-task feature standardization (proposal review): the project does not assume arbitrary SAE features are aligned across tasks. We use a shared 128-dim policy trunk, mean-pooled episode aggregation, and a pooled grid+Lunar SAE dictionary, then evaluate calibrated transfer directly (report §7.4). A target-domain validation set is needed to resolve sign polarity, as documented in the Grid → Lunar sign-flip discussion.
• SAE sparsity (check-in review): the final experiments raise the L1 penalty rather than switching to Top-K. Mean L0 falls from ~377/1024 at the midpoint to ~35/1024 in the final gridworld SAE, while in-domain test AUROC remains at 1.000 (report Table 2).
• Statistical comparison (final pass): all transfer AUROC numbers carry 95% stratified-bootstrap confidence intervals, and SAE-vs-Raw differences carry paired-bootstrap p-values (report §7.3 and §7.4).
• Causal feature evidence (final pass): report §7.5 adds single-feature ablation experiments. A single honest-selective gridworld feature (941) accounts for 0.41 of the 0.49-above-chance AUROC margin in cross-seed transfer.

Setup

pip install -e .
pip install -e ".[learning]"   # torch, numpy, scikit-learn, matplotlib, gymnasium[box2d]

The gymnasium[box2d] dependency (required for LunarLanderContinuous-v3) needs swig as a system-level build tool:

# macOS
brew install swig

# Ubuntu/Debian
sudo apt-get install swig

Testing

PYTHONPATH=src python -m unittest discover -s tests

Notes:

• The existing tests are focused on the gridworld environment logic.
• Importing hackrl currently also imports the LunarLander module, so running tests that touch package imports may require the optional gymnasium[box2d] dependency even if you only care about gridworld.