This repository studies self-play systems in which search is both a source of supervision and an object of learning.
The basic ratchet is:
network
↓
PUCT search
↓
better policy, Q, and root-value targets
↓
distillation
↓
stronger network
↓
stronger future search
V5.8 extends that loop by learning a model of search computation itself:
student state
↓
shared hidden-computation representation
├── should this root receive more search?
├── which legal branch should receive it?
└── will another simulation block add information?
↓
adaptive PUCT teacher
↓
policy/Q/value distillation
No expert games, human move labels, handcrafted value function, or shaped reward are used. The only external consequence is the terminal result:
[ z \in {-1,0,+1}. ]
The current 6×6 connect-3 suite compares adaptive V5.8, fixed-budget V5.8, MuZero-Q, and ordinary MuZero. Every run used seed 1337, stopped at 10,000 optimizer steps, trained on 1,280,000 optimizer examples, and was evaluated with a fixed 16-simulation search budget.
| Variant | Configuration | Seed | Step | Training wall time (s) | Fixed-budget Elo | Fixed-budget score |
|---|---|---|---|---|---|---|
| V5.8 adaptive | 1e4cf04e2c37 |
1337 | 10,000 | 41,975.251 | 1,813.147 | 0.996647 |
| V5.8 fixed | 41f4f1b0c9f5 |
1337 | 10,000 | 28,666.908 | 1,810.804 | 0.997301 |
| MuZero-Q | 62e1d49dbceb |
1337 | 10,000 | 26,500.236 | 1,789.943 | 0.995526 |
| MuZero | c8e57838f5db |
1337 | 10,000 | 30,010.966 | 1,665.385 | 0.980337 |
| Variant | Self-play states | Search simulations | Optimizer examples | Peak RSS |
|---|---|---|---|---|
| V5.8 adaptive | 163,194 | 3,038,888 | 1,280,000 | 4,535,304,192 bytes (4.22 GiB) |
| V5.8 fixed | 164,192 | 3,569,184 | 1,280,000 | 4,186,947,584 bytes (3.90 GiB) |
| MuZero-Q | 195,846 | 4,255,936 | 1,280,000 | 4,735,139,840 bytes (4.41 GiB) |
| MuZero | 227,580 | 4,966,512 | 1,280,000 | 4,170,219,520 bytes (3.88 GiB) |
At the measured endpoints, both V5.8 variants reached the 1810-Elo region. Adaptive V5.8 used about 17% fewer self-play states and 29% fewer search simulations than MuZero-Q, while ordinary MuZero finished materially weaker. These are single-seed endpoint measurements rather than universal claims; repeated seeds and confidence intervals are still required.
The Transformer consumes a canonical player-to-move meta-state:
[BOS] [current board] [SEP] [recent autoregressive action history]
It predicts:
- a policy over legal actions;
- a bounded Q value for every legal action;
- a latent state used by the population and computation systems.
The scalar value is derived from the legal policy and Q field:
[ V(s)=\sum_a \pi(a\mid s)Q(s,a). ]
For every legal action, search confidence is
[ c(a)=\frac{N(a)}{N(a)+\lambda_Q}. ]
The target blends backed-up search evidence with the Q field recorded before search:
[ Q_{target}(a) =c(a)Q_{PUCT}(a)+(1-c(a))Q_{prior}(a). ]
Visited actions receive stronger search supervision. Unvisited legal actions retain a lower-weight temporal anchor rather than being treated as if search had evaluated them.
Shared hidden-computation representation
V5.8 does not use three unrelated controllers for root allocation, branch geometry, and stopping. They are projections of one shared representation of remaining computation.
The action-conditioned recursive quantity is summarized as
[ H(s,a)=I(s,a)+\gamma H(s'), ]
with the search-policy expectation
[ H_\pi(s)=\sum_a\pi_{search}(a\mid s)H(s,a). ]
The target is decomposed into epistemic uncertainty and proof difficulty so the system can distinguish “the model does not know” from “the continuation remains expensive to establish.”
The computation model also learns an immediate action cost and predicted successor cost. Its cost recursion is
[ C(s)=\min_a\left[c(s,a)+\gamma C(s'\mid a)\right]. ]
An exponential-moving-average target copy stabilizes successor targets. The predicted contraction is
[ G(s,a)=\max\left(0,C(s)-\gamma C(s'\mid a)\right). ]
When calibration and replay readiness permit, training search uses
[ \operatorname{Score}(s,a) =Q(s,a)+U_{PUCT}(s,a)+\lambda(s,d)G(s,a). ]
The trust-conditioned (\lambda) is bounded. Fixed-budget evaluation and disabled-controller ablations remain ordinary (Q+U_{PUCT}) search.
One tree is reused across the cumulative simulation ladder 8/16/32/64. At each boundary, V5.8 records how much the next block changes policy, root value, and legal-action Q:
[ M(s,n)\approx JS(\pi_{n+\Delta n},\pi_n) +\alpha|V_{n+\Delta n}-V_n| +\beta\operatorname{mean}a|Q{n+\Delta n}(a)-Q_n(a)|. ]
Full-ladder exploration supplies counterfactual labels. Once the Bellman system is ready, predicted remaining computation controls the ladder ceiling; the marginal head remains a calibration measurement.
V5.8 retains the earlier homeostatic machinery:
- latent-neighbour population occupancy;
- C1 control of neighbourhood bandwidth;
- Virtual D prediction of delayed population structure;
- C2 control of virtual-signal authority;
- C3 delayed supervision of interacting controller consequences;
- temporal information, forecast, assimilation, and frontier diagnostics.
These systems remain separate from terminal game recursion. Terminal outcomes propagate backward through game trajectories, while population forecasts and controllers operate across optimizer time.
Collection mixes:
- 40% current-policy mirror play;
- 40% recent frozen league opponents;
- 20% promoted hall-of-fame opponents.
Frozen opponents change the states reached but do not provide stale policy or Q labels. Only the live side's searched turns enter replay. The first league snapshot is permanent, and candidate promotion is confidence-gated.
At league evaluation, budgets 0, 16, 32, and 64 are tested from checkpointed fixed openings with deterministic action selection and root noise disabled. This separates capability already compressed into the raw policy from strength still supplied by search.
The lower-cost research harness. Its default is 6×6 connect-3 and it contains the complete V5.8 learner, game, replay, search, league, controllers, checkpointing, and dashboard.
An independent full implementation, not a wrapper around the TTT script. Its default environment is full-size 19×19 Go:
- Black first and White receives 7.5 komi;
- capture by removal of groups without liberties;
- suicide prevention;
- positional superko for placement actions;
- pass as action 361 after board actions 0–360;
- termination after two consecutive passes or a 722-ply safety cap;
- Chinese area scoring;
- eight rotation/reflection symmetries, with pass invariant;
- 362 policy/Q actions;
- full-board input plus the latest 149 action tokens.
This is a self-play research environment. It does not implement handicap setup, resignation, clocks, dead-stone negotiation, or Japanese territory/prisoner scoring.
Python 3.10 or newer is recommended.
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install numpy torch matplotlib
# V5.8 TTT benchmark
python3 ttt_v5_8.py
# Independent full-size 19x19 Go experiment
python3 go_v5_8.pyThe scripts require no CLI configuration. Pressing the editor's Play button is
equivalent to running the file. Edit the Config dataclass for experiment
changes.
Device selection is automatic: CUDA, then Apple MPS, then CPU.
Full-size Go is substantially more expensive than the TTT harness. Its default batch size, gradient accumulation, replay probes, evaluation games, and collection rate are reduced for the larger action and context spaces. The algorithmic search ladder remains intact.
Every experiment has its own artifact root:
runs_ttt_v5_8/ttt_v5_8/config_<fingerprint>/
runs_go_v5_8/go_v5_8/config_<fingerprint>/
Trajectory-changing configuration values create a new fingerprinted subdirectory. Operational changes such as plotting frequency do not fork the trajectory lineage.
Within each active configuration directory:
checkpoint_latest.ptis atomically replaced every 100 optimizer steps;- the checkpoint stores complete resumable state, including model and target models, optimizers, replay, controllers, league, diagnostic banks, and RNG;
- league snapshots remain under the separate
league/directory; - CSV histories and the dashboard are restored rather than rewritten on resume;
latest.txtpoints to the active configuration directory.
Generated experiment directories and checkpoint files are excluded by
.gitignore.
Never load an untrusted full-state PyTorch checkpoint.
The reported Elo is meaningful only inside its configured evaluation protocol. It is not a human, online-server, or professional Go rating.
Useful evidence should distinguish:
- self-play states from optimizer updates and total simulations;
- raw-policy strength from search-supplied strength;
- live-root assimilation from fixed-root frontier migration;
- immediate search correction from persistent downstream contraction;
- state-space coverage from matched-strength sample efficiency;
- a single endpoint from repeated-seed confidence intervals.
The strongest next benchmark is a fixed-computation comparison in which all methods receive the same total simulations and are evaluated at matched policy strength. The current result is encouraging because V5.8 reaches the same approximate strength region as MuZero-Q with materially fewer self-play states, but it should be treated as a measured experiment rather than a settled general law.
The progression is:
V3 learn population structure
V4 regulate optimization dynamics
V5 distill search into policy and Q
V5.2 learn whether to think
V5.3 learn where to think
V5.4 learn when to stop
V5.5 share a hidden-computation representation
V5.8 learn action-conditioned recursive computation
MuZero learns game value while using search as a fixed inference procedure. This line of work asks whether a second policy can learn the allocation and future value of computation itself:
The game policy learns what to play. The computation policy learns where thinking is worth spending.