Inequity Aversion Improves Cooperation in Intertemporal Social Dilemmas — Hughes et al. (2018)

This is a from-scratch paper reproduction, not a ported codebase. This repo rebuilds Hughes et al. (2018)'s Cleanup and Harvest environments and its inequity-aversion reward mechanism directly from the paper, with its own independent actor-critic training and no dependency on Vinitsky et al.'s code. For an engineering port of Vinitsky's existing sequential_social_dilemma_games codebase to Ray RLlib's new API stack instead — which already includes a simpler, instantaneous-reward version of this same mechanism as an opt-in flag (inequity_averse_reward) — see the sibling repo SequentialSocialDilemmas. For the from-scratch reproduction of the earlier DeepMind SSD paper (Leibo et al. 2017, Gathering and Wolfpack, no Cleanup/Harvest at all), see Leibo2017. Three repos with overlapping lineage and easy-to-confuse names — this note, and matching notes in the other two, are here so none of them gets mistaken for another.

This repo does not currently reproduce the paper's headline Cleanup claim, and Harvest is inconclusive. Cleanup: advantageous_only (guilt) does not beat baseline on collective return at the scale tested here, the opposite of the paper's Fig. 3A. Harvest: mean collective return is a near-tie across all three conditions at only 3 seeds, though disadvantageous_only is notably more consistent seed-to-seed than the other two, suggestively (not confirmed) in line with the paper's own Fig. 4 story. See "Known gaps from the paper" below for the full results and what's been ruled out.

A from-scratch replication of:

Hughes, E., Leibo, J. Z., Phillips, M., Tuyls, K., Dueñez-Guzmán, E., Castañeda, A. G., Dunning, I., Zhu, T., McKee, K. R., Koster, R., Roff, H., & Graepel, T. (2018). Inequity Aversion Improves Cooperation in Intertemporal Social Dilemmas. NeurIPS 2018. https://arxiv.org/abs/1803.08884

The paper's question: purely self-interested reinforcement learners tend to fail sequential social dilemmas like Cleanup and Harvest (the individually rational move — never clean, always harvest — collapses the shared resource for everyone). Fehr & Schmidt (1999)'s inequity-aversion model explains a lot of human cooperative behavior in one-shot economic games with a simple idea: people don't just want reward, they're also averse to inequity — doing worse than others (envy) and, to a lesser extent, doing better than others (guilt). This paper asks whether giving that same aversion to independent reinforcement-learning agents, extended from a single-payoff comparison to a reward stream comparison, produces the same cooperation-enabling effect in temporally extended, spatial social dilemmas — and finds that it does. Cleanup and Harvest, both introduced in this paper, later became the standard testbeds this whole line of DeepMind work builds on (Vinitsky et al.'s port, and McKee et al. 2023's reputation experiment reused in the sibling SequentialSocialDilemmas repo, both use these same two environments).

The mechanism

Each agent's actual training reward is r_i^total = r_i - inequity_penalty_i, where each agent tracks a discounted trace of its own reward (Eq. 4, verified against arXiv v3):

e_i(t) = gamma * lambda * e_i(t-1) + r_i(t)                          (every step)

inequity_penalty_i = (alpha_i / (n-1)) * sum_j max(e_j - e_i, 0)     # envy
                    + (beta_i  / (n-1)) * sum_j max(e_i - e_j, 0)    # guilt

alpha_i (envy weight) and beta_i (guilt weight) are sampled once per agent at construction — population heterogeneity, not resampled per episode, matching the same design used for the reward term in the sibling SequentialSocialDilemmas repo's cleanup_reputation experiment. e_i is not a bounded running average (an earlier version of this repo used one -- e <- lambda*e + (1-lambda)*r -- before catching the discrepancy against the primary text in review): it's an unnormalized, discount-accumulating trace using the same gamma as the RL return, so for a roughly constant reward rate it settles near r / (1 - gamma*lambda), several times larger than the raw reward itself, not the same scale.

The one thing this repo gets right that the simpler port doesn't: SequentialSocialDilemmas' inequity_averse_reward compares agents' raw, instantaneous per-step rewards. The paper is explicit that this is too noisy to be meaningful — Fehr-Schmidt's original model compares final payoffs, and the natural temporally-extended analogue is a discounted trace of the reward stream, not a single noisy step (one agent happening to step onto an apple this exact tick, another not, says nothing about who's actually "ahead"). e_i above is that trace (InequityAversionReward in hughes2018/reward/inequity_aversion.py) — see test_matches_hand_computed_trace_formula in tests/test_inequity_aversion.py for the exact arithmetic, verified against a hand-computed example.

The paper additionally has each agent observe every other player's trace e_j as part of its own policy input, so an agent can react to inequity, not just be shaped by it as an invisible reward signal — see "What's matched vs. simplified" below for how this repo now does the same (trace_dim/observe_trace, opt-in, applied uniformly across conditions including baseline).

What's matched vs. simplified

Matched, from the paper's own stated design:

• Both environments' core dynamics: Cleanup's piecewise-linear apple-regrowth-vs-river-pollution curve (THRESHOLD_DEPLETION=0.4, THRESHOLD_RESTORATION=0.0, WASTE_SPAWN_PROBABILITY=0.5, APPLE_RESPAWN_PROBABILITY=0.05) and Harvest's local-density-dependent regrowth (SPAWN_PROBABILITY_BY_NEIGHBOR_COUNT = (0.0, 0.005, 0.02, 0.05) indexed by apple count within an L1 (Manhattan) radius of 2 — Sec. A.3: "apples spawn relative to the current number of other apples within an l1 radius of 2") — these are the paper's own environment mechanics (independently corroborated by Vinitsky et al.'s port and McKee et al. 2023's reuse of the same Cleanup design, not copied from either). APPLE_RESPAWN_PROBABILITY specifically: the paper's own Sec. A.3 prose says "probability 0.125x", but two independent reference implementations of this exact environment both use 0.05 instead and agree with the paper's text on every other checkable constant: DeepMind's own dmlab2d (simulation.lua's appleRespawnProbability) and Eugene Vinitsky et al.'s independent, paper-contemporaneous port (social_dilemmas/envs/cleanup.py's appleRespawnProbability) — two sources agreeing with each other over a hand-written sentence is strong enough to call the paper's own prose the error here. This repo went 0.05 (original) → 0.125 (paper prose, wrong turn) → 0.05 (dmlab2d, then confirmed again by Vinitsky's port, current) across successive review passes. (Not to be confused with Melting Pot's own clean_up substrate, a different, later reimplementation that hardcodes 7 players — not the paper's N=5 — and a timeout-based punishment beam with zero reward penalty, i.e. the older mechanic this paper explicitly contrasts itself against; it cites Hughes et al. 2018 as the environment's origin but isn't a faithful reproduction of this paper's own setup, so it isn't used as evidence anywhere in this repo. dmlab2d's avatar_list.lua, by contrast, takes numPlayers as an external parameter rather than hardcoding it, consistent with being closer to the original paper's own configurable setup.) An earlier version of this repo also used a Moore/Chebyshev-radius-1 neighborhood for Harvest instead of the paper's stated L1-radius-2 — caught by reading the primary source's Sec. A.3 parameter appendix directly, not by inspection; not flagged by code review, since it was internally consistent, just a fidelity gap against a number the paper states outright.
• Beam cooldowns and a post-reset waste-growth grace period were added, then reverted. dmlab2d's reference implementation rate-limits both beams (avatar.lua's fineWait=10/cleanWait=2) and delays new waste accumulation for the first 50 steps of each episode (simulation.lua's dirtGrowthStartTime) — neither mentioned in the paper's own prose. Both were added here on that evidence alone. But unlike APPLE_RESPAWN_PROBABILITY above, there was no second source to corroborate dmlab2d: Vinitsky et al.'s independent, paper-contemporaneous port has neither mechanic — no cooldown state anywhere in its agent.py/map_env.py, and custom_map_update() spawns waste unconditionally every step. One source for, silence-or-absence everywhere else, is a materially weaker case than the apple-probability one. It's also the kind of change that's easy to convince yourself is real once you've found it in one authoritative-looking place — worth naming plainly rather than glossing over. Reverted in favor of staying consistent with Vinitsky's port, which this repo now trusts as the more load-bearing reference for the paper's unstated details (it's also independently confirmed the flat action space and the 15×15 window below). Concretely motivating the revert: a 3-seed baseline vs. advantageous_only comparison was a statistical tie before these two mechanics were added, and a clean, consistent baseline-wins-3-of-3 result after — suspicious enough on its own to warrant backing out a weakly-evidenced change rather than trust a result entangled with it.
• Cleanup's episode-reset waste level: "the environment resets with waste just beyond this saturation point" (Sec. 2.4) — RESET_WASTE_DENSITY=0.42, just past THRESHOLD_DEPLETION=0.4, not the whole river. (An earlier version of this repo filled the entire river with waste at reset instead — density 1.0 — requiring far more cleaning before any apple could possibly spawn than the paper's own design; caught by reading the primary source directly, not by inspection.)
• Advantageous vs. disadvantageous inequity aversion tested separately for Cleanup, matching the paper's own Fig. 3: "(A-C) compares... A3C and advantageous inequity averse agents... (D-F) demonstrate that disadvantageous inequity aversion does not promote greater cooperation in the Cleanup game." run_experiment1_cleanup.py runs three conditions — baseline, advantageous_only (guilt, alpha=0 — the condition the paper shows actually helps Cleanup), and both (this repo's original combined condition, kept for reference but not one of the paper's own two tested Cleanup conditions). An earlier version of this repo only ever tested the combined condition, which conflates the paper's two separately-tested and differently-behaving mechanisms; caught by reading the primary source, not by inspection.
• The flat, mutually-exclusive action set (move forward/backward/strafe-left/strafe-right relative to current facing, turn left/right, stay, a beam — one choice per step) and 3-cell-wide, 5-cell-long beam geometry. Cross-checked specifically against dmlab2d's own reference implementation, which instead uses a factored, combinable action space (move × turn × fireClean × fireFine as independent dimensions settable simultaneously, ~60 total combinations) — but the paper's own footnote 3 states "|A^i| ranges from 8 to 10", a single small count per agent, which only makes sense for a flat space (this repo's NUM_ACTIONS_BASE=8/NUM_ACTIONS_CLEANUP=9 match exactly); nobody would describe a ~60-combination factored space that way. So dmlab2d's factored action space is judged to be a later dmlab2d/Melting-Pot engineering convenience, not what Hughes et al. 2018's own code used — this repo's existing flat design is the better-supported one and was deliberately not changed to match dmlab2d here. Vinitsky et al.'s independent port agrees: a flat DiscreteWithDType(9, ...) action space, no factoring.
• The punishment beam is a fine (target loses 50 reward, shooter pays 1), not a timeout/removal — the paper explicitly contrasts this with the earlier SSD literature's timeout-based beam. (An earlier version of this repo implemented timeout-based removal instead; caught in review against the primary text.) Also cross-checked against dmlab2d: its beam geometry (hitBeam(piece, type, length=5, radius=1)) matches this repo's own 3-wide/5-long beam exactly, and its -1/-50 reward split matches too.
• The 15×15 centered observation window (_VIEW_RADIUS_DEFAULT=7), matching the paper's own footnote 3 ("d = 15 × 15 × 3"). dmlab2d's reference implementation instead uses an 11×11, off-center, forward-biased window (left=5, right=5, forward=9, backward=1, centered=false) — but that directly contradicts the paper's own stated number too, not just this repo, so (unlike the apple-probability case, where dmlab2d agreed with the paper everywhere else) there's no reason here to prefer dmlab2d's number over the paper's own explicit footnote. Judged to be a dmlab2d/Melting-Pot-specific choice; deliberately not adopted. Vinitsky et al.'s independent port agrees with the paper here too: CLEANUP_VIEW_SIZE=7, used as a radius (2*view_len+1 in map_env.py's observation-space shape) — the same centered 15×15×3 this repo uses.
• The network architecture explicitly stated (by McKee et al. 2023's Materials and Methods) to be inherited from this paper's own setup: 3x3-kernel/32-channel conv, two 64-unit FC layers, a 128-unit LSTM, linear policy/value heads.
• Per-agent independent actor-critic learners, no parameter sharing, no communication — the paper's stated independence assumption.
• The inequity-aversion formula's shape (a discounted trace, envy/guilt terms normalized by n-1) — see "The mechanism" above for the correction from an earlier, incorrectly-normalized version.
• Agents observe every player's inequity trace, not just their own reward. Sec. 3.2: "we allow agents to observe the smoothed reward of every player on each timestep" — describing the inequity-averse agent's own design, not the unmodified A3C baseline it's compared against, which has no trace concept at all. Opt in via ActorCriticConfig(trace_dim=num_agents) (the default path) or --observe-trace/observe_trace=True (the RLlib backend, on by default); run_experiment1_cleanup.py/run_experiment2_harvest.py and the RLlib backend all enable it for baseline too (an always-constant-zero, informationless trace vector), so the network architecture — not just the reward — stays identical across a comparison. Causality matters here: the trace paired with a given action is always the trace as of the previous step, never one that already reflects that action's own outcome — see InequityAversionReward.observable_trace()'s docstring. trace_dim=0 (the default when not explicitly requested) keeps the network and observation exactly as before. (The RLlib backend's baseline case had a real bug here, not just a design nuance: an earlier version built a live InequityAversionReward(alpha=0, beta=0) tracker for baseline "for architectural uniformity" — its .trace is computed by apply()'s decay*trace + r, which runs independent of alpha/beta, so even at alpha=beta=0 it was a real, informative smoothed-reward signal, not a no-op. That gave baseline a real observation channel the paper's own A3C baseline never had, while advantageous_only got that same information plus a real penalty on top — biasing any comparison toward baseline regardless of whether guilt actually helps. Caught only empirically: baseline consistently outperformed advantageous_only across a 3-seed comparison, the opposite of the paper's own Fig. 3 result, until this was found and fixed.)
• Exact map layouts for Cleanup and Harvest. Neither paper publishes pixel-exact level files. hughes2018/envs/cleanup.py/harvest.py use their own map designs with the right qualitative structure (a spatially separate river and orchard for Cleanup; an open apple field for Harvest), not a copy of any existing implementation's specific ASCII map.
• Synchronous n-step actor-critic instead of asynchronous A3C. The paper trains with true A3C (Mnih et al. 2016): multiple worker processes computing gradients independently and asynchronously against a shared parameter server. This repo collects one fixed-length rollout at a time and updates each agent's own network synchronously — same independent-per-agent actor-critic method, simplified execution model. See hughes2018/agents/actor_critic.py's module docstring.
• The inequity-aversion trace's decay parameter (lambda). The paper states the trace uses gamma * lambda decay but doesn't give an exact numeric value for lambda itself. InequityAversionReward.trace_lambda (default 0.95) is a documented, tunable parameter, not a verified reproduction of an unstated number. gamma defaults to 0.99, matching the actor-critic's own discount factor, per the paper's reuse of the same gamma.
• Learning rate, entropy coefficient, and alpha/beta sampling ranges are not quoted from the paper's primary text here; the alpha ~ U(2.4, 3.0), beta ~ U(0.16, 0.20) default used in the run scripts is carried over from the same range used for a related reward term in the sibling SequentialSocialDilemmas repo's cleanup_reputation experiment (itself following the general shape this lineage of papers typically uses), not a citation-backed number from this specific paper. Note these ranges were calibrated for a bounded-average-scale trace in an earlier version of this repo; now that the trace matches the paper's unnormalized, larger-magnitude formula, these defaults may need retuning for sensible training dynamics — not yet verified at real training scale.
• Movement-conflict resolution. Two agents can't occupy the same cell; conflicts are resolved by processing agents in a random per-step order (an agent moves into its target cell only if unoccupied at that moment, otherwise stays put). This is simpler than Vinitsky's multi-pass algorithm (which additionally lets two agents swap places in one step) and isn't specified by the paper either way.

Running it

git clone git@github.com:doesburg11/Hughes2018.git
cd Hughes2018
./create_conda_env.sh
conda activate ./.conda
# or: pip install -r requirements.txt into any Python 3.11 environment

pytest tests/ -q

Every run_experiment*.py script defaults to a small step count that exercises the full pipeline (env → independent actor-critic training → inequity-aversion reward → comparison) as a smoke test — it does not claim converged, paper-scale results. Pass --total-steps with a much larger value (and patience, or a GPU via --device cuda) to attempt that.

python run_experiment1_cleanup.py     # Cleanup: baseline vs. advantageous-only (paper's condition) vs. both
python run_experiment2_harvest.py     # Harvest: baseline vs. inequity-averse
python run_experiment3_heterogeneous.py   # mixed population: some inequity-averse, some selfish

Each writes results.json (and, if matplotlib is available, a comparison plot) to output/<script-name>/, and now also saves each trained agent's weights to output/<script-name>/checkpoints/<condition>/<agent-id>.pt (network weights only, not optimizer/LSTM state -- a checkpoint here is for evaluation, not resuming training).

Watching a trained policy

render_rollout.py plays out one episode and renders it to an animated GIF, using the env's own third-person render() frame (upscaled with nearest-neighbor resizing so cells stay crisp):

python render_rollout.py --env cleanup                                                     # random policy
python render_rollout.py --env cleanup --checkpoint-dir output/run_experiment1_cleanup/checkpoints/baseline
python render_rollout.py --env harvest --checkpoint-dir output/run_experiment2_harvest/checkpoints/inequity-averse

Writes to output/render_rollout/<env>_<random|trained>.gif by default (override with --out).

Optional: RLlib backend

Everything above trains with hughes2018/agents/actor_critic.py and hughes2018/training/loop.py — a single-process, synchronous n-step actor-critic loop, and the only thing every run_experiment*.py script and the README's fidelity claims are about. run_rllib_train.py is a separate, purely additive way to train the exact same CleanupEnv/HarvestEnv with Ray RLlib's PPO or IMPALA instead, using RLlib's own env-runner pool for real parallel rollout collection across multiple CPU cores — the paper's own setup uses 24 asynchronous copies per agent, which this repo's hand-rolled training loop doesn't attempt, and which a from-scratch multiprocessing implementation would mostly be re-solving problems RLlib already solves (see hughes2018/envs/rllib_wrappers.py's module docstring). It does not touch hughes2018/agents/actor_critic.py or any existing script, and isn't required for anything else in this repo — the base pip install -r requirements.txt has no Ray/RLlib dependency at all.

pip install -r requirements-rllib.txt   # ray[rllib]; tested against ray==2.58.0

# Cleanup, IMPALA (closest RLlib algorithm to the paper's own asynchronous
# actor-critic with staleness correction -- V-trace), 8 parallel env-runners,
# advantageous-only inequity aversion (the paper's own tested Cleanup condition):
python run_rllib_train.py --env cleanup --algorithm IMPALA \
    --num-env-runners 8 --condition advantageous_only --iterations 200

# Baseline, no inequity term, saving a checkpoint:
python run_rllib_train.py --env cleanup --condition baseline --iterations 200 \
    --checkpoint-dir output/rllib_checkpoints/cleanup_baseline

--condition supports the same baseline / advantageous_only / disadvantageous_only / both split as run_experiment1_cleanup.py (see "What's matched vs. simplified" above), sampling alpha/beta via the same scaled_alpha_beta_range() helper. --num-env-runners controls parallelism (tune to available CPU cores); --rollout-length sets the LSTM's truncated-BPTT window (RLlib's max_seq_len — required once use_lstm=True, since RLlib has no default for a stateful RLModule); --gpu puts the (tiny) network's training on GPU.

Documented simplifications specific to this backend (on top of the ones already listed above for the default path — see hughes2018/envs/rllib_wrappers.py's module docstring for the full reasoning):

• Observations are flattened to a 1D vector rather than kept as a (C, H, W) image, and the model config relies on fcnet_hiddens + use_lstm only, no explicit conv layer — RLlib's Catalog auto-detects 3D Box observations as images but has no preset for this environment's non-standard channel-first shape (the same issue the sibling SequentialSocialDilemmas repo's own RLlib adapter documents and sidesteps the same way). The default path's network does include the paper-matching 3x3/32-channel conv layer; this is a further, RLlib-specific simplification on top of that.
• Reaching episode_length is reported as a truncation, not a termination — more correct than training/loop.py's own treatment (documented there as a known simplification), since it's a time-limit cutoff, not a true terminal state.
• Not a path to reproducing the paper's own headline result at its own scale by itself: it's untested above small smoke-test scale (a handful of env-runners, a few agents, tens of iterations) as of this writing — running it larger is the natural next step, not something already verified here.

Known gaps from the paper

• This repo does not currently reproduce the paper's headline Cleanup claim. The paper's Fig. 3A: advantageous_only (guilt) should out-perform baseline A3C on collective return. Here, across every configuration tested, baseline wins instead. The default (non-RLlib) training path makes no attempt at the paper's actual training scale (see "Running it" above); the optional RLlib backend has been run at and beyond the paper's own stated Cleanup scale (Fig. 3's ~1.6-1.8M agent steps; runs here reached ~4.2M) across four rounds of a 3-seed baseline vs. advantageous_only vs. disadvantageous_only comparison, tracking this repo's own fixes as they landed:

  1. With the trace-observability bug above still present: baseline won 2 of 3 seeds by ~7.5 points on average — a systematic gap the wrong way round.
  2. After fixing that bug: a 1-1 tie, gap down to ~2 points, within each condition's own seed spread — genuinely inconclusive.
  3. After also adding beam cooldowns and the waste grace period (both since reverted — see above): a clean baseline-wins-3-of-3 result re-emerged (mean 21.8 vs. 13.5) — which is what motivated reverting those two mechanics, since the evidence for them was weak and this was the second time a weakly-evidenced change happened to flip the result away from the paper's claim.
  4. On the reverted, Vinitsky-consistent code: baseline won 3 of 3 seeds again (mean 11.9 vs. 7.0, disadvantageous_only mean −185.8) — so backing out that change did not explain the discrepancy away. baseline has now beaten advantageous_only in 6 of 6 individual seed comparisons across two materially different environment configurations.

  disadvantageous_only collapses far below both in every round, consistent with the paper's own claim there (Fig. 3D-F) — it's specifically the advantageous/guilt result that doesn't reproduce. Investigation was stopped at this point (2026-09-09) rather than pursued further; this is the current honest state, not a resolved question. Ruled out along the way: under-training (2.5x the paper's own budget), beta being too weak (a sweep showed larger beta strictly hurts), single-seed noise (consistent 3/3 and 6/6 patterns), and the cooldown/grace-period mechanics (reverted, result persisted). Not investigated: IMPALA's V-trace correction vs. the paper's literal asynchronous A3C as a genuine algorithmic difference (discussed but not tested in isolation), a finer alpha/beta sweep below the values already tried, and a from-scratch architecture audit beyond what's covered above. Whoever picks this up next should re-verify the numbers above still match the current code before trusting them — training runs, not something pytest covers.

• Harvest is inconclusive, not confirmed or refuted. Unlike Cleanup, Harvest had never been seriously trained before 2026-09-09 — only smoke-tested for plumbing. Two checks first, before any real training: (a) cross-referenced Harvest's mechanics against Vinitsky et al.'s independent port — the regrowth probability table [0, 0.005, 0.02, 0.05], the 15×15 window, the flat Discrete(8) action space, and a full-at-reset orchard all matched exactly, so no environment fixes were needed going in; (b) one real discrepancy found and not acted on: Vinitsky's actual neighbor-counting code (j**2+k**2 <= 2) resolves to a Moore/Chebyshev-radius-1 neighborhood (8 cells), not the L1-radius-2 diamond (12 cells) the paper's own text explicitly states — the mirror image of the Cleanup apple-probability situation, but without a second source to corroborate overriding the paper's prose (dmlab2d's own commons_harvest uses a third, different probability table and radius entirely, confirming it's an unrelated, generically-reused environment, not calibrated to this paper — kept out of the comparison). Then one round of the same 3-seed baseline/advantageous_only/disadvantageous_only comparison, well short of the paper's own Harvest scale (Fig. 4: ~1.0×10⁸ agent steps for the advantageous panels, ~7×10⁷ for the disadvantageous panels — both roughly 20-30x further than the ~3.6-3.8M reached here, unlike the Cleanup runs which matched or exceeded the paper's own Cleanup budget): mean collective return was a near-tie across all three (baseline 2184, advantageous_only 2209, disadvantageous_only 2464), each condition winning 1-2 of 3 seeds against the others — not separated at this sample size. The one notable pattern: disadvantageous_only's seed-to-seed spread (±98) was far tighter than baseline's or advantageous_only's (±1075, ±1027) — a consistency difference, not just a mean difference, loosely in the direction of the paper's own Fig. 4 claim that disadvantageous inequity aversion "works even when only 1 out of 5 agents are inequity averse." Suggestive, not confirmed — 3 seeds isn't enough to call this a real effect, and the run didn't reach the paper's own training scale the way the Cleanup runs did. Stopped here (2026-09-09), same as Cleanup — documented as the current honest state, not pursued further.

• run_experiment3_heterogeneous.py is a first-order version of the paper's population-heterogeneity question (do inequity-averse agents get exploited by selfish co-players in a single fixed-composition training run), not its full evolutionary/generational-selection treatment across which trait survives over many generations — that's a documented follow-up, not built here.

• No convolutional-architecture ablation or hyperparameter sweep reproducing the paper's own robustness checks (if it has any beyond the headline Cleanup/Harvest comparison — not verified against the primary text in this pass).

Tests

tests/ covers, in order of dependency: the shared gridworld primitives (movement, rotation, beam geometry, observation orientation — the same "sign verified against beam/move directions" rigor the sibling Leibo2017 repo uses for its own primitives), each environment's paper-specific mechanics (Cleanup's pollution/regrowth curve, Harvest's density-dependent spawning), the inequity-aversion reward formula (including a hand-computed exact-value check against the trace formula), the actor-critic agent (including a hand-computed discounted-return check, and a check that every named parameter updates after a gradient step — the kind of check that catches a layer silently excluded from the optimizer, which a weaker "did any parameter change" check missed in an earlier version), an end-to-end training smoke test, and a fake-env/fake-agent regression test verifying the training loop pairs each stored observation with the action actually chosen from it (not the next step's observation). Run with pytest tests/ -q.

Acknowledgments

Developed with AI coding assistance from Claude (Anthropic), which does the implementation, with Codex (OpenAI) acting as an independent second opinion, peer-reviewing Claude's nontrivial code changes.

References

• Hughes, E., Leibo, J. Z., Phillips, M., Tuyls, K., Dueñez-Guzmán, E., Castañeda, A. G., Dunning, I., Zhu, T., McKee, K. R., Koster, R., Roff, H., & Graepel, T. (2018). Inequity Aversion Improves Cooperation in Intertemporal Social Dilemmas. NeurIPS 2018.
• Fehr, E., & Schmidt, K. M. (1999). A Theory of Fairness, Competition, and Cooperation. The Quarterly Journal of Economics, 114(3), 817-868.
• Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., & Kavukcuoglu, K. (2016). Asynchronous Methods for Deep Reinforcement Learning. ICML 2016. (The A3C algorithm this paper trains with, and this repo's own actor-critic simplifies.)
• McKee, K. R., Hughes, E., Zhu, T. O., Chadwick, M. J., Koster, R., Castañeda, A. G., Beattie, C., Graepel, T., Botvinick, M., & Leibo, J. Z. (2023). A Multi-Agent Reinforcement Learning Model of Reputation and Cooperation in Human Groups. arXiv:2103.04982. (Reuses this paper's Cleanup design; source of the network-architecture spec cited above.)
• Vinitsky, E., Jaques, N., Leibo, J., Castañeda, A., Hughes, E., et al. Sequential Social Dilemma Games — the open-source port this project's sibling repo, SequentialSocialDilemmas, updates to Ray's new API stack.