ml_skull

Multi-agent reinforcement learning for the bluffing game Skull. Agents are trained via self-play PPO on a PettingZoo environment wrapping a pure-Python game engine.

Layout

env/
  game_state.py       # Pure game logic (stdlib only)
  obs_encoder.py      # State -> observation vector + action mask
  skull_env.py        # PettingZoo AECEnv wrapper
training/
  train_selfplay.py   # RLlib + PPO + self-play (--restore to continue a run)
  evaluate.py         # Load a checkpoint; win rate vs random / another checkpoint
  heuristic_agent.py  # Rule-based "safe-bidder with planted skull" benchmark
  duel.py             # Symmetric head-to-head: A vs B (random|heuristic|checkpoint)
tests/
  test_random.py      # Random-agent sanity check
  test_game_state.py  # Direct unit tests of the game engine
skull_ruleset.txt     # ML-friendly ruleset (source of truth)
CLAUDE.md             # Project conventions

Setup

pip install pettingzoo "ray[rllib]" torch gymnasium

Usage

python -m pytest tests/             # run the test suite
python tests/test_random.py         # sanity check the environment
python training/train_selfplay.py   # start self-play training
python training/evaluate.py         # evaluate latest checkpoint vs random
python training/duel.py checkpoints/iter_0300 heuristic --stochastic  # head-to-head

To continue a finished run from its last checkpoint:

python training/train_selfplay.py --num-iters 300 --restore checkpoints/iter_0300

duel.py runs a symmetric match (equal seats per side, rotated over all balanced seat combinations) so neither side gets a positional edge — use it to compare two checkpoints, since the lone-seat evaluate.py format is confounded when both sides are trained policies. heuristic_agent.py is a fixed benchmark tougher than random (it beats random ~5:1).

Checkpoints are written to checkpoints/ every 25 iterations, and the training loop now prints win rate vs random at each checkpoint (tune with --eval-every / --eval-games, or --eval-every 0 to disable) so the vs-random curve is visible during training, not just episode_return_mean. evaluate.py does the same standalone for any checkpoint, rotating the trained agent through every seat (cancelling first-player bias) against random opponents or --opponent <checkpoint>.

Game

2–6 players, each with 4 discs (3 flowers + 1 skull). Players place discs face-down, then bid on how many they can flip without hitting a skull. The highest bidder flips that many — own stack first, then opponents' in any order. Flipping a skull loses a disc; succeeding flips the mat. Two mat flips wins the game.

The ML challenge is partial observability + bluffing: opponent stack contents are hidden and must be inferred from betting behavior.

Environment

• Observation (per agent): public state (disc counts, stack sizes, mat orientations, current bid, pass flags) + own hand and own stack contents. Opponent stack contents are never exposed.
• Actions: phase-dependent, flattened to a single Discrete(ACTION_SPACE_SIZE) with an action_mask — PLACE(disc), BID(n), RAISE(n), PASS, FLIP(player_id).
• Rewards: +1 win, +0.3 succeed challenge, -0.3 fail, -0.1 lose disc, +0.2 eliminate opponent, -1 eliminated.

Training roadmap

1. Random baseline — verify uniform win rates and no illegal states.
2. Self-play PPO via RLlib (current).
3. Belief module over hidden opponent discs (optional).
4. CFR via OpenSpiel for stronger bluffing behavior (optional).

See CLAUDE.md for conventions and skull_ruleset.txt for the full rules.