BlockBlaster

An end-to-end Monte Carlo value agent that plays Block Blast on a real phone

Train a value network in simulation. Recognise the live game from a mirrored screen. Drive the finger via a closed-loop visual servo. Watch the phone play itself.

Demo

The agent picks moves, the visual servo lands them on a real Android device. Full-resolution clip: assets/Blockblaster_Demo.mp4.

---

What this actually is

Most "Block Blast AI" projects stop at search heuristics or a paper plot of MC returns. This repo runs the whole loop end-to-end:

flowchart LR
    subgraph TRAIN["**1. Train v(s)**"]
        direction TB
        T1["Simulator<br/>(pure-Python)"]
        T2["CNN value head"]
        T3["Monte Carlo returns"]
        T4["Potential-based<br/>reward shaping"]
        T5["D4 symmetry<br/>augmentation"]
        T1 --> T2 --> T3 --> T4 --> T5
    end

    subgraph PERCEIVE["**2. Perceive game**"]
        direction TB
        P1["scrcpy mirror<br/>(live frames)"]
        P2["Board / queue scanner<br/>(HSV + grid)"]
        P3["Piece classifier CNN<br/>(synth-data trained)"]
        P4["Assist GUI overlay<br/>+ ghost recon panel"]
        P1 --> P2 --> P3 --> P4
    end

    subgraph ACT["**3. Act on the device**"]
        direction TB
        A1["Advisor → suggestion"]
        A2["Visual servo drag<br/>(P controller)"]
        A3["Per-frame template match<br/>(piece silhouette)"]
        A4["Release when locked"]
        A1 --> A2 --> A3 --> A4
    end

    TRAIN ==> PERCEIVE ==> ACT
    ACT -. closed loop .-> PERCEIVE

    classDef stage fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#e2e8f0
    class TRAIN,PERCEIVE,ACT stage

Loading

Each stage is a real, runnable subsystem — not a notebook stub. The agent that decides moves is the same agent that plays in simulation; the perception pipeline that draws overlays is the same one that feeds the action loop; the servo that drags pieces closes its own loop on every frame off a per-frame template match.

Why the pieces are interesting

The value network is a small CNN trained via Monte Carlo returns with potential-based reward shaping (Ng, Harada & Russell 1999) and D4 symmetry augmentation of board states. The agent acts greedily on afterstate value with the shaping potential added back at decision time so the optimal policy is unchanged. See docs/algorithm.md.

The perception stack is built on a tiny piece-classifier CNN trained entirely on synthetic data — every queue tile, every render variant, generated on the fly. The assist GUI overlays the agent's planned move on the mirrored screen in real time and includes a reconstructed-scene panel that lets you see what the scanner sees (placed cells, ghost preview, queue confidences). See docs/assist-gui.md.

The visual servo is the bit most projects skip. It closes the loop on the device: every frame, diff the board crop against the pre-grab snapshot to get a motion mask, diff again against the previous frame as a glow-resistant translation gate, template-match the held piece's silhouette within a small window around the last trusted location, and PD-step the finger toward the matched anchors until the error and the match score are both inside their tolerances — then lift.

Quick start

uv sync                                          # install everything
uv run simulate.py                               # collect MC episodes
uv run train.py                                  # fit v(s) on the dataset
uv run main.py                                   # watch the trained agent play in-sim
uv run play.py --platform ios --mode assist      # live overlay on a mirrored iPhone
uv run play.py --platform android                # full auto-play on Android

The Android path is the headline feature: connect a phone over ADB, launch Block Blast, run the command, and the agent will calibrate the board, pick moves, and drive the finger via the closed-loop servo.

Repository tour

Path	What lives there
blockblaster/game/	Pure-Python Block Blast simulator: pieces, board, scoring, legal-move generation.
blockblaster/model/	State encoder and value-network architecture.
blockblaster/agent/	Decision policy: afterstate enumeration, value lookup, beam-search lookahead.
blockblaster/piece_cnn/	Synthetic data generator and CNN that classifies queue tiles from pixels.
blockblaster/assist/	Pygame assist GUI, screen analyzer, board/queue scanner, recon panel, advisor wiring.
blockblaster/control/	Device abstractions (ADB, scrcpy), servo.py closed-loop placer, calibration.
docs/	Long-form documentation per subsystem (see below).

Documentation

The README is just the index. Each doc cross-references the others, so any one of them is a reasonable entry point depending on what you came for.

Doc	Read it for
docs/game-rules.md	Board / queue / scoring rules and the 42-piece enumeration.
docs/architecture.md	Top-level folder layout and per-subpackage maps.
docs/algorithm.md	State encoding, value network, Monte Carlo pipeline with beam-search lookahead, potential-based reward shaping, D4 augmentation, champion / challenger checkpointing.
docs/policy.md	How select_action works: 3-piece beam search, full-return scoring (Σ γ^k r_k + γ^K V*(leaf)), distinct-orderings handling, dead-end semantics, future improvements.
docs/sim-configs.md	Named default vs quality simulation presets — which params to flip for fast-iteration vs trustworthy data, and why.
docs/hyperparameters.md	Every knob in param.py with its default and meaning.
docs/training.md	simulate → train → repeat loop, generated files, how to watch the trained agent play.
docs/assist-gui.md	Side-by-side viewer, calibration flow, key bindings, the synthetic-data piece classifier.
docs/android-autoplay.md	Emulator / physical-phone setup, scrcpy v1.20 + adbutils touch tunnel, end-to-end auto-play.

Status & limitations

Honest about what works and what doesn't:

• Simulation pipeline: stable. Train, evaluate, watch in-sim.
• iOS assist (read-only overlay): works on a mirrored iPhone — pure visualisation, no input injection (Apple doesn't allow it without a paired Mac/Xcode signature).
• Android auto-play: working but device-specific. The servo's diff threshold and match-score tolerances may need a small retune for very different board palettes; the geometry-related knobs (step sizes, error thresholds, search radius) are expressed as cell ratios and resolve from the calibrated grid automatically. See blockblaster/config/params.py.
• Calibration: semi-manual on first use — drop the grid + queue boxes on the mirrored frame once, persisted to JSON for subsequent runs.

If you fork this and play with a different game, the perception + control split is reusable: the servo doesn't know anything about Block Blast specifically, only about "drag this finger so that thing on screen lines up with that target."