Hangzhou Mahjong with Public-Information Transformers and Paired Branch Reinforcement Learning

This project provides a Hangzhou Mahjong agent, its training and evaluation framework, and results from a frozen internal opponent pool. Hangzhou Mahjong combines hidden information, variable-length matches, persistent dealer status, and multiplicative scoring. These properties make decisions depend on both local hand structure and public table history. The 52.01-million-parameter policy combines a tile-relation Transformer, a causal event encoder, legal-action queries, public tactical successor features, and compact opponent memory. The reported checkpoint inherits an already trained policy, receives supervised action fine-tuning, and continues with paired branch reinforcement learning. Each eligible decision on a reference trajectory produces five exploratory continuations. Their realized scores are compared with a reference over the same game window, and an empirical clipped objective updates the exploration distribution using its recorded temperature and tactical bias. Evaluation rotates complete tables through all four seats and reports a fixed table-length-weighted net score.

In a completed 36-participant tournament comprising 1,081,008 games, with 120,112 seat-games per participant, the released r2600 checkpoint ranked first under the predefined weighted score, at +0.452171 per seat-game. The next participant scored +0.430258; their difference was +0.021914, with an exploratory 50% bootstrap interval of [−0.024367, +0.069381]. The interval does not establish a reliable separation between the two. This repository contains the training framework, game simulator and evaluator, and the selected r2600 weights. It supports running the released policy and evaluating user-supplied checkpoints. Historical rule opponents, other neural weights, and the demonstration corpus are not included.

Keywords: imperfect-information games; Mahjong; reinforcement learning; causal memory; structured exploration; policy optimization.

The objective is to maximize net score in a four-player local Hangzhou Mahjong environment. Winning frequency alone is insufficient: the payoff of a win varies with the hand pattern, action chain, and dealer relationship. A policy must also account for how its current decisions affect the next round and future dealer status.

The report documents three engineering contributions, without claiming that their individual benefits have been experimentally isolated:

1. A public-information policy. Hand structure, legal actions, public events, the player's own past decisions, and completed-round summaries are represented separately and fused by attention.
2. Paired short-horizon exploration. Alternative continuations start from the same simulator state as a reference policy, providing a realized comparison without using a learned value function as the branch baseline.
3. A reproducible tournament protocol. Frozen weights, fixed random plans, four-seat rotations, equal participation, and explicit table-length weights define the reported score.

The simulator is a particular Hangzhou rule set described below. Results are conditional on these rules and the evaluated opponent pool. The project does not claim universal Mahjong strength, human-level performance, or an ablation-proven contribution from each component.

The empirical question answered here is narrow: how do the frozen participants rank under one specified tournament distribution and score? The tournament does not test whether this training method outperforms another method under matched data and compute. The contribution is a concrete system definition, a runnable training/evaluation framework with one checkpoint, and the conditional tournament result.

The architecture and learning objective address different parts of the decision problem:

These are implementation choices and their motivations. The final tournament evaluates the complete system; it does not isolate the causal effect of each choice.

Suphx [1] is a relevant reference for organizing a Mahjong system around its decision flow, representations, learning signals, and evaluation. It studies Riichi Mahjong and combines supervised initialization with reinforcement learning. Its reported methods include global reward prediction, oracle-guided training, and policy adaptation during play. The following comparison concerns mechanisms, not playing strength (see [1], Sections 2–5).

Public look-ahead features are another connection to Suphx's representation design. Here they are encoded as action-owned successor vectors and bounded one-/two-draw estimates; they do not constitute a full search over hidden states. Different rules, objectives, and evaluation populations preclude a numerical strength comparison with Suphx. This work reports its own architecture and observed tournament results without transferring Suphx's human-performance claims to the released model.

Simulation-based rollout methods use a base policy to estimate the consequences of candidate choices and connect this procedure to policy improvement [6]. This system instead samples short exploratory continuations during training and fits a parametric policy with a clipped loss. It does not perform exhaustive rollout action selection during evaluation, and the improvement conditions in [6] do not establish an improvement guarantee for this objective.

PEGASUS [7] studies policy search with a deterministic simulative model and fixed random scenarios. Our pairing similarly reuses a root state and wall-seed plan to reduce uncontrolled differences between a branch and its reference. We do not implement the PEGASUS policy-search procedure or establish its assumptions here. Pairing is not a proof of variance reduction in this environment: different actions can change tile consumption and downstream events.

COMA [8] addresses cooperative multi-agent credit assignment through a centralized critic and a baseline that marginalizes one agent's action. Our reference is a realized deterministic continuation in a competitive, zero-sum game; it is neither that critic nor an action-marginalized counterfactual baseline. The clipped probability-ratio update is inspired by PPO [2], but uses shared continuation returns, stale sampled policies, and outcome-dependent weights. These distinctions matter when interpreting the method as an empirical system design rather than a theoretical policy-gradient result.

The following units are used throughout; the word round in source-code field names can denote either a game or an optimizer iteration, so those units are made explicit in the text.

The environment uses 136 tiles: four copies of each of 34 tile types. White dragons serve as wild tiles. Only self-drawn wins are allowed; robbing a kong is not part of this rule set. The final 20 wall tiles are reserved. A kong requires a replacement tile to remain available. A player may make at most two chi melds.

Discarding a wild tile starts a restricted catch-and-discard cycle. During that cycle, the other players cannot chi, peng, or claim an exposed kong, and their discard is restricted to the drawn tile; concealed/additional kongs and self-drawn wins remain subject to their own legality checks. The simulator handles the response priority and action sequencing.

The fan multiplier is

$$ F=H,2^c,2^{I_{\mathrm{four\ white}}},2^{I_{\mathrm{baotou}}}. $$

The factors stack. For example, an ordinary baotou win after a two-action chain, without the four-white bonus, has fan $1\times4\times2=8$. Natural quads in seven pairs are hand-pattern bonuses, distinct from declared kongs. The experiment disables the optional restriction that would require a player holding whites to win only through baotou or a kong replacement; ordinary self-drawn wins while holding whites are allowed. Fan is multiplicative, not an additive score.

For winner $w$, loser $j$, dealer $d$, base unit $b=1$, and fan $F$,

$$ p_{j\rightarrow w}=bF \begin{cases} 8,&j=d\ \text{or}\ w=d,\\ 1,&\text{otherwise}. \end{cases} $$

Each of the other three players pays the winner separately. Thus each game's score changes sum to zero. The winner becomes the next dealer; a draw leaves the dealer unchanged. A match contains $N$ games. Branch training uses lengths from 8 to 16: each block of six tables contains one 8-game table, one 16-game table, and four tables whose lengths are drawn uniformly from 9 through 15. The main evaluation instead covers $N\in{1,\ldots,18}$, with the scoring weights defined in Section 6.

At decision time, the policy receives its own hand, public discards and melds, public action events, scores, dealer and round context, and legal actions. It does not receive the wall order or the other players' concealed hands.

There are three distinct uses of information:

Auxiliary targets are training labels, not policy inputs. Branching clones the simulator state to execute alternative outcomes; the model still observes only its own view. This distinction is necessary when interpreting the paired-return estimate: it is a simulator-based training signal, not access to hidden information during play.

The selected checkpoint contains 52,005,062 parameters. It uses the query action head, dynamic match-length conditioning, and public tactical memory.

The current board is represented by 34 tile-type tokens, each with 101 input channels. Separate tokens represent pooled board context and the current action request. The request combines the action kind, requested tile, relative source seat, match position, and a 202-dimensional context containing scalar context and the legal-action mask.

Self-attention over tile tokens includes learned relation biases. Relations distinguish identical tiles, within-suit rank offsets, equal ranks across suits, wild-tile relationships, and other tile pairs. This supplies a structured inductive bias while leaving the learned policy responsible for the action decision.

Residual blocks use RMS normalization [3] and SwiGLU feed-forward transformations [5]. The model does not run a game-tree search at deployment.

Public event tokens and the player's own decisions form a causal sequence. Event features include token identity, public side information, tile references, and the player's initial hand at the start of each round. Own-decision records encode the observation available when that decision was made and the submitted action.

The history encoder uses rotary position embeddings [4]. The state stream reads the permitted history prefix through gated cross-attention after state blocks 4, 10, and 16, using one-based numbering. An action query also reads history before its final score is produced.

Training assembles complete table histories with a separate visible-prefix boundary for every decision. A decision cannot attend to future public events or future decisions from the same table. At inference, incremental key/value caches avoid repeatedly encoding an unchanged prefix. Training recomputes history representations under the current model parameters and propagates gradients through them.

flowchart LR
    S[Current public observation] --> T[Tile-relation Transformer]
    H[Public events and own decisions] --> C[Causal history Transformer]
    C --> T
    M[Public opponent profiles and completed rounds] --> T
    A[Legal action candidates] --> Q[Action queries]
    X[Public tactical successor features] --> Q
    T --> Q
    C --> Q
    Q --> P[Masked action scores]

Actions occupy a fixed 142-element space:

Only legal candidates are assembled as action queries. A query combines the request representation, target tile, supporting tiles, and action type. It reads the current state and permitted history, then produces a scalar score. Invalid actions are masked. Index 142 is used internally for batch padding and is never an executable action.

When an exposed-kong response also allows peng, the implementation composes two conditional distributions:

$$ P(\text{kong})=P_g(\text{accept}),\quad P(\text{peng})=P_g(\text{pass})P_p(\text{accept}),\quad P(\text{pass})=P_g(\text{pass})P_p(\text{pass}). $$

Both conditional predictions use the same historical cutoff. Rejecting the kong does not fabricate an extra historical event or training decision.

Legal candidates can lead to multiple meaningful local successor states. The feature builder constructs these successors using only the public observation and the acting player's hand. It represents ordinary/seven-pairs progress, improving tiles, structurally winning tiles, publicly unseen counts, legal winning opportunities, immediate fan and income, meld status, action-chain state, and match context.

A successor vector contains 71 scalar features with availability indicators and 17 tile-wise fields over 34 tile types:

$$ d_{\text{successor}}=2\times71+17\times34=720. $$

The availability channels distinguish an unknown or inapplicable quantity from a genuine zero. Count/distance quantities use fixed units; selected score and fan quantities use logarithmic encoding.

The feature builder also contains bounded local one-/two-draw approximations. These estimates use public unseen-tile counts and hand-shape calculations, not the actual future wall. They are features for the policy and exploration mechanism; they are not calibrated probabilities of winning an entire game.

Each candidate query attends only to successor rows owned by that candidate. Consequently, successor representations remain aligned with the action whose consequences they describe.

Opponent memory is updated from public events and completed settlements. It contains cumulative statistics, a bounded recent-event summary, smoothed discard distributions, and availability/context metadata. Each of the three opponents has a 193-dimensional profile. Completed rounds produce 304-dimensional summaries. A summary becomes visible only after its round has finished.

A separate reader attends to at most the latest 32 history events from the current round. This supports local response context without replacing the full causal history stream.

For round index $t\in{0,\ldots,N-1}$, a 10-anchor embedding uses the continuous position

$$ u_t=\frac{9t}{\max(N-1,1)}. $$

The neighboring anchor vectors are linearly interpolated. A single-game match uses the final anchor explicitly. Additional continuous horizon features condition the current-state and history streams on match length and remaining horizon.

The released checkpoint was not trained from scratch in the final fine-tuning run. Its immediate parent was an already trained policy at iteration 2537. Four epochs of action-only supervised fine-tuning supplied the adopted weights, after which branch RL continued through iteration 2600. The RL continuation retained the parent RL optimizer state, not the supervised optimizer. The iteration number is a lineage label, not a count of all optimization used to produce the model.

The first four supervised epochs took 377.29 seconds in aggregate including their setup and saves, excluding data preparation and model-selection games. This is the compute on the adopted path; the seven-epoch search and its evaluation cost extra. Epoch 4's 90.59% action agreement is a training statistic, not held-out accuracy. The RL quantities above are sums of recorded per-batch counts; raw table identifiers were not independently deduplicated in this summary.

During the final RL segment, three opponents were sampled uniformly with replacement from a 25-entry pool: 12 frozen older neural models, 12 rule policies, and the earlier current-family r1500 checkpoint. The learner's seat varied with the table schedule. This is substantial overlap between training opponents and final evaluation participants, not an unseen-opponent benchmark. The reference and five branches for a packet use the same frozen learner version. Sampling ahead of learning produced observed lag values of 0–3 completed iterations relative to the learner state immediately before its next update; stored behavior probabilities are used by the loss.

The inherited parent itself came from prior supervised and RL training. This report does not reconstruct its complete pre-2537 data/compute budget or every earlier model-selection decision. Therefore four supervised epochs plus 63 RL iterations is only a documented final training segment, not the total cost of obtaining r2600. Neither the demonstration corpus nor all ancestral checkpoints are included. The final 36-participant tournament is also a checkpoint-selection result, not an untouched confirmation set.

The supplied general-purpose supervised implementation accepts complete-table datasets with action labels or weighted teacher action distributions. Its optional features below are framework capabilities; they must not be read as additional objectives used in the action-only fine-tuning stage above. Let $y_t(a)$ be a target distribution over legal actions and $w_t$ its validity/importance weight. The policy term is

$$ L_{\mathrm{policy}}= -\frac{\sum_t w_t\sum_a y_t(a)\log\pi_\theta(a\mid o_{\leq t})} {\max(1,\sum_t w_t)}. $$

An optional value regression target and auxiliary count-classification targets support representation learning:

$$ L_{\mathrm{sup}}= L_{\mathrm{policy}}+\lambda_v L_{\mathrm{value}} +\frac{\lambda_{\mathrm{aux}}}{2} \left(L_{\mathrm{opponent\ counts}}+L_{\mathrm{wall\ counts}}\right). $$

The auxiliary head predicts five count categories, 0–4, for each tile type, covering three opponents and the wall. These labels remain outside the inference input.

The trainer groups data by complete tables, supports separate training/validation partitions, and optionally permutes the three suits consistently across observations, actions, masks, and history. It supports BF16 computation, activation checkpointing, distributed data parallelism, and resumable model/optimizer/random-state checkpoints.

The historical demonstration corpus and its preprocessing inputs are not included. The supervised trainer is supplied for compatible user-provided data; the selected weights cannot be regenerated from this repository alone.

Each sampling batch contains $G=12$ separately seeded reference tables. A table fixes its lineup, random wall plan, initial dealer, match length, and policy random streams. The reference learner follows the deterministic policy.

An eligible learner decision has more than one legal action. The simulator state and policy history are captured immediately before that decision. Duplicate callbacks for an already-resolved conditional meld response do not create extra roots. From each root, the sampler executes five separately sampled exploratory continuations. They have separate action random streams but share the root and reference, and their direction choices are partly coupled. They are not five statistically independent samples.

Exploration and learning decisions apply only through the remainder of the current game. At its end, exploration is cleared; the immediately following game, if present, is played with deterministic learner actions for settlement only. The return covers at most these two games and stops at the end of the table. This permits a limited effect from the next dealer state without rolling out to the end of every match.

Algorithm 1 — One paired-branch training iteration

1. Fix a behavior checkpoint, frozen opponent pool, and sampling seeds.
2. For each of 12 tables:
   a. Sample three opponents with replacement and fix seats, dealer,
      table length, and the per-game wall plan.
   b. Run the deterministic reference table and save eligible learner roots.
   c. At each root, construct five exploration profiles (Section 5.2).
      For each profile, clone the root, sample learner actions until the
      current game ends, then settle one further game deterministically
      if the table has another game. Opponent policies stay fixed.
   d. Store current-game learning decisions, their behavior probabilities
      and biases, and branch/reference scores over the same game window.
3. Compute one paired return and outcome multiplier for each continuation.
4. Randomly split each table's roots into two disjoint halves.
5. Process each half once with the stored behavior distributions and the
   weighted clipped objective; make one optimizer update per half.

Multiple roots can share reference history and continuation outcomes. Counts of branch decisions therefore measure optimization workload, not independent games or independent statistical evidence.

For branch $b$, legal action $a$, policy score $z_\theta(a)$, temperature $T_b$, and bounded tactical rank bias $B_b(a)$,

$$ q_{\theta,b}(a\mid o)= \operatorname{softmax}_{a\in\mathcal A(o)} \left(\frac{z_\theta(a)}{T_b}+\lambda_b B_b(a)\right). $$

For the exposed-kong/peng compound response in Section 3.3, $z_\theta(a)$ is the log of the composed action probability. Temperature and tactical bias are applied after composing the two conditional predictions, not separately to each conditional distribution before multiplication. The ordinary action path uses its masked model scores.

The reference configuration accompanying r2600 is:

The distribution is also conditioned on a direction vector and a local plan state; these arguments are suppressed in $q_{\theta,b}(a\mid o)$. Families are $A=(A1,A2,A3)$, $B=(B1,\ldots,B5)$, $C=(C1,\ldots,C4)$, and $D=(D1,\ldots,D5)$. Branch 3 samples a family uniformly, then a direction uniformly within it. Branch 4 excludes that family before drawing its own. Branch 5 samples two distinct families and one direction from each, mixing their weights with $\alpha\sim U(0.25,0.75)$ and $1-\alpha$. B1 and B3 are excluded for roots with declared melds. If the learner subsequently melds, their weights transfer to A1. Profiles persist within a game; events update plans rather than redrawing a direction at each decision.

Direction scores. All quantities below come from a public hypothetical successor $x$, not the true wall. Let $u_t$ be unseen copies of tile type $t$, $U=\max(1,\sum_tu_t)$, $d_o,d_7$ the ordinary/seven-pairs hand distances, and $I_o,I_7$ their improving-tile counts weighted by $u_t$. The ordinary distance includes the implementation's false-ready correction; the seven-pairs route is disabled after a meld. Let $H$ be the number of structural winning tile types, $L$ their unseen-copy total, $g$ the held white count, $p$ the current-chain piao count, and $c$ the chain length. Define

$$ f_o=\frac{8-\operatorname{clip}(d_o,0,8)+I_o/136}{9},\qquad f_7=\frac{8-\operatorname{clip}(d_7,0,8)+I_7/136}{9} $$

with $f_7=I_7=0$ for a melded hand. Let $Q$ count natural non-white quads, $J$ count non-white triplet types with unseen copies, and $P$ count natural non-white pairs (a quad counts twice). Put $b_3=\min(1,(Q+J/4)/3)$ for a closed hand and zero otherwise, $Z=\min((g+p)/4,1)$, $C=\min(c,4)/4$, and $W=H/34$. The material score is

$$ m=\max\left{\frac{f_o+Z+W+C}{4},; \frac{f_7+b_3+Z+W+C}{5};\text{if closed, else }0\right}. $$

For each hypothetical draw, $h_t$ indicates a structural wait, $e_t$ indicates a legal self-drawn win, $F_t$ is its fan, and $Y_t$ is its immediate receipt under the current dealer. Let $Y_{\min}$ be the minimum receipt over publicly possible legal winning draws. Write $v_{32}=\sum_tu_te_t\min(F_t/32,1)/U$ and $\phi(y)=\min(\log_2(1+y/10),8)/8$.

A3 is calculated only when at least one nonterminal candidate successor is within one distance step of ready; otherwise it is zero. These bounded local estimates are heuristics and do not assign probabilities to opponent responses. Terminal successor route scores are zero before plan adjustments.

An action can own several hypothetical successors, for example a meld followed by each legal discard. For direction weights $\omega_b$ and plan-adjusted score vector $\widetilde f(x)$, its scalar preference is

$$ s_a=\max_{x\in\mathcal X(a)}\omega_b^\top\widetilde f(x). $$

This is an optimistic maximum over the listed public successors, not their average and not a full game-tree value. The bounded rank transform below acts on these action-level preferences.

Finite continuation plans. At a win decision, ordinary direction scores are cleared: only active B5/D3/D4 plans may bias accepting or declining the win. A possible upgrade must have a remaining draw, at least one unseen tile type, legal wins for every publicly possible next draw, and a minimum next-win receipt strictly above the current receipt. Non-kong continuations must additionally retain structural waits on all 34 tile types. These checks concern the learner's local continuation and do not guarantee reaching the next draw before an opponent wins.

White-discard continuations require baotou. Remaining ties use the lower action identifier. If more than one of these directions has positive weight, the planner checks B5, D3, then D4 in that order. An active plan supplies a 1/0 score for the requested plan action in its direction coordinate. Other plan-only coordinates are zero; B5 retains its material score only outside the win gate and outside a B5 plan.

The progression is decline win → execute the selected discard/kong → wait for a real draw or replacement → accept the win. It advances on actual selected actions and observed events, not on repeated queries. B5 reevaluates after arrival; D3/D4 cash at the following win gate. A deviation that invalidates the plan clears it, and game end clears all plans and direction weights. Bias changes sampling probabilities; it never forces the planned action or guarantees the projected payoff.

For candidate scalar preferences $s_a$, the rank transform is

$$ B(a)= \frac{#{a':s_a>s_{a'}}-#{a':s_a<s_{a'}}} {|\mathcal A|-1}, $$

with zero bias for a single candidate. Its magnitude is bounded by one, and ties receive the same rank score. The mechanism biases sampling rather than overriding action legality.

Every sampled decision stores its action, behavior log probability, temperature, bias vector, and bias strength. Learning holds the recorded bias and profile settings fixed while reevaluating the policy logits for the probability ratio. Deterministic evaluation uses the legal policy argmax without exploratory temperature or tactical sampling bias.

Let $R_{gkb}$ be the learner's realized net score in branch $b$ of root $k$ in table $g$. Let $R^{\mathrm{ref}}_{gk}$ be the reference score over the same round-index window, and let $N_g$ be the table length. The branch advantage is

$$ A_{gkb}= \frac{R_{gkb}-R^{\mathrm{ref}}_{gk}}{64N_g}. $$

This single advantage is assigned to the learner's trainable decisions in that exploratory continuation. The comparison uses the same initial root and wall-seed plan, although different actions can change the realized draw and action sequence.

Here “advantage” names a realized continuation-score difference, shared by multiple decisions. It is not an unbiased estimate of the conventional state-action advantage $A^\pi(s,a)$ at every sampled decision. The divisor 64 is a numerical scaling constant, not a bound on possible Mahjong scores; $N_g$ normalizes by table horizon even though the return window is shorter.

There is no learned critic baseline in this branch objective. The value and auxiliary heads are frozen. This reduces dependence on critic calibration, but it does not eliminate variance, short-horizon bias, or stale-policy effects from sampling ahead of the learner.

With stored behavior probability $q_{\mathrm{old}}$,

$$ \rho_t=\frac{q_{\theta,b}(a_t\mid o_t)} {q_{\mathrm{old},b}(a_t\mid o_t)}. $$

The per-decision empirical loss uses the clipped PPO form [2], with entropy regularization on the actual sampling distribution:

$$ \ell_t= -\min!\left(\rho_t A,\operatorname{clip}(\rho_t,0.8,1.2)A\right) -\frac{\beta}{N_g}H(q_{\theta,b}(\cdot\mid o_t)), \qquad \beta=0.001. $$

For each root/branch pair, the maximum fan won by the learner in either its exploratory or reference return window determines a sample multiplier:

This is a training importance heuristic, not an importance correction for a naturally sampled game population. A high-fan reference win can therefore weight a negative exploratory branch even when that branch does not win.

Roots within each table are randomly divided into two disjoint halves. Each half produces one optimizer step; a root is not trained twice merely because the batch uses two steps. If table $g$ has $K_g$ roots, the implementation assigns each decision the base factor

$$ c_g=\frac{2}{100,G,\max(1,K_g),5}. $$

The half-batch loss sums $c_g W_{gkb}\ell_t$ over its decisions. There is no additional division by the number of decisions in a branch. Thus table/root normalization and decision count play different roles, and longer continuations can contribute more gradient terms.

The factor 2 compensates for using half the roots in each update, while 100 sets the overall loss scale. Together with sign-dependent, outcome-dependent $W$, this defines an empirical surrogate objective. It is not mathematically equivalent to maximizing the full-table evaluation score. In particular, $W$ reweights return outcomes, and improving a temperature-and-bias sampling policy $q$ need not improve the plain argmax deployment policy. This report makes no monotonic-improvement or unbiased-policy-gradient claim for the combination.

The branch update uses AdamW with zero weight decay, learning rate $6\times10^{-5}$, epsilon $10^{-9}$, and global gradient norm clipping at $0.001$. The two learner GPUs, when enabled, partition microbatches and reduce gradients into a single optimizer state.

The learning process and sampling processes are separated. Samplers host frozen models and reusable inference services; the learner consumes complete branch packets. A bounded pipeline supports two batches being computed and an in-flight depth of four. Behavior versions remain attached to sampled data.

Within a root, immutable history prefixes and their inference caches can be shared across branches. The learner reconstructs the necessary raw prefix/suffix data and recomputes differentiable representations. Atomic branch-training checkpoints retain the complete model, optimizer, configuration, and run state, including the scheduling seed and round counter used to derive sampling streams. They do not capture the complete Python, NumPy, and Torch random-number-generator states; this differs from the supervised trainer's checkpoint format.

The reference experiment used two learner GPUs and two eight-GPU sampling machines. These describe the experiment's execution environment, not a minimum hardware requirement. The public entry point also supports a single learner GPU and user-specified sampler devices. No GPU instance or remote hostname is embedded in the release.

An evaluation fixes model identities, rule implementations, a seed, lineups, and match lengths before collecting results. Identical neural weights are deduplicated even if supplied under different names.

A group contains four rotations of a complete $N$-game table. Random streams for lineup selection, walls, and policy seeds are separate. The same table plan is reused across rotations, and each participating lineup position visits every seat. Scores and dealer transitions persist within each table.

The schedule equalizes participant seat-games and seat positions. It also improves opponent-pair balance by swapping scheduled lineup positions. Same-version copies are allowed at a table; it is not an exhaustive round-robin of every possible four-player combination.

For participant $i$, let $\mu_{i,N}$ be total net score divided by its seat-game count among tables of length $N$. The official score is

$$ S_i=\sum_{N=1}^{18}w_N\mu_{i,N},\qquad w_N= \begin{cases} 0.75/7,&10\le N\le16,\\ 0.25/11,&\text{otherwise}. \end{cases} $$

Weights apply to table length, not to fan, winning hands, or a post hoc classification of games. A participant's raw cumulative score is separately reported as the unweighted sum of its actual score changes. These summaries can rank participants differently.

The 75% mass on 10–16-game tables is a preset design preference for medium-length matches while retaining coverage of shorter and longer ones. It is not an estimate of the frequency of match lengths in a measured user population, and no external calibration of the specific 75% choice is available. The plan fixed these weights before results were collected. The result therefore answers a question about this chosen utility, rather than identifying a universally strongest policy.

The evaluator requests a target participation count and rounds up to satisfy the mirrored schedule. Actual counts, rather than requested counts, are used in denominators.

Bootstrap resampling is stratified by table length. The resampling unit is a complete four-seat mirrored group; games within that group are not treated as independent observations. The implementation performs 10,000 resamples and retains the original leading-pair central 50% interval.

The leading pair is selected from the observed ranking. This is an exploratory comparison, not a simultaneous confidence guarantee over all participants or an independent confirmation experiment. In particular, a 50% interval is weaker evidence than a 95% interval.

For fan $f\in{8,16,32}$, the reported rate is

$$ \operatorname{rate}_{i,f}= \frac{#{\text{participant }i\text{ wins at exactly fan }f}} {#{\text{all participant }i\text{ seat-games}}}. $$

A 64-fan win is not counted as a 32-fan win. These are observed event frequencies, not conditional opportunities under fixed hypothetical hands. Games are not divided into supposedly comparable “ordinary” and “high-fan” populations based on their eventual outcomes.

The completed tournament contained 36 participants: 24 neural checkpoints and 12 rule policies. It comprised 23,364 complete mirrored groups, 1,081,008 games, and 120,112 seat-games per participant, with 30,028 in each seat. Its frozen plan used seed 2026101003; reproducing the original games would also require its exact participant assets and implementation identities.

This is an internal pool of checkpoint and policy identities, not 36 independently developed algorithms. In the table below, v5 denotes the current model family; hof marks a retained checkpoint from an earlier selection. Numbered checkpoints are not matched-compute ablations. In particular, hof_r003400 and v5_r003400 are different weights from different training lineages.

The neural router participant is a direct action model trained on routed behavior, not a test-time ensemble of the twelve rules. The retained teacher-routing implementation selects a rule style at the player's first action request in a game, using the current public state and history through the game-start boundary, and keeps that style for the game. The neural model learns action outputs from routed behavior. Historical supervision records distinguish the mixed-teacher base objective, routed-action labels, and individual-teacher labels; the tournament does not isolate their respective training effects. Full ancestry and aggregate compute for every older checkpoint are not reconstructed here.

The rule pool includes speed-oriented FastHu, fan-building FanBuild/HighScore, pair-oriented PairBuild/PairKeep/PairHold, meld/near-ready policies PMH/MeldNearReady, baotou-oriented BaotouForecast, wall-tempo WallTempo, and two pre-existing hangma rule variants. These are descriptive categories, not standardized external benchmarks. The _cash wrapper checks for a same-turn kong replacement that preserves a win on every publicly possible replacement tile and strictly improves its minimum fan, then cashes the win. It retains the underlying style's other choices. A neural entry and a rule: entry sharing a name are separate participants, with potentially different actions and scores.

Only the framework and r2600 weights are released. These descriptions and aggregate results do not imply that the historical opponents are included or that their whole training provenance is publicly reproducible.

The selected r2600 checkpoint ranked first by the official weighted score. All 36 results are shown; positive and negative cumulative scores use the same zero-sum settlement units as Section 2.

Participant identifiers denote distinct checkpoints, not additional models bundled with this release. The comparison is a single final same-pool tournament, not a claim about a chronological training trend.

The official first-minus-second difference was +0.021913517, with the original 50% interval [−0.024367052, +0.069381479]. The leading checkpoint is selected by the predefined score; the interval does not establish that it is consistently stronger than r2800. Raw cumulative score instead favors r2800.

The original per-length score summaries give the following descriptive decomposition. Each column averages the already reported per-seat-game means equally across the listed table lengths; it is not a new tournament or bootstrap.

r2600's higher overall point estimate comes from the other lengths; it is lower on the emphasized 10–16 segment. Replacing the two group weights by $\alpha$ and $1-\alpha$ would make these two point estimates equal at $\alpha\approx0.802125$, favoring r2800 above that value. This is arithmetic on existing means, not evidence of a statistically reliable difference within either length group. The official weights and ranking remain unchanged. No new interval or alternative primary result is inferred from this sensitivity calculation.

Selected exact-fan results are:

Every percentage uses 120,112 seat-games as its denominator. The r3300 and r3400 participants each recorded one 64-fan win, separately from these exact-fan counts.

All group identifiers were complete and unique; participation and seat counts matched the schedule. Re-aggregation reproduced the official ranking, with a maximum floating-point mean discrepancy of $1.67\times10^{-16}$. The aggregate net score over all participants was zero. These are result-integrity checks, not independent replications or additional evidence of playing strength.

The public package includes only r2600's neural weights and no historical opponent implementations or additional weights. Comparative evaluation requires users to supply their own compatible checkpoints. Reconstructing the original tournament would require the remaining weights, all twelve rule policies, and its original frozen plan. A new run with a different pool cannot be compared numerically to the table above as if it were the same experiment. Single-checkpoint self-play is available only as a software check; it cannot measure relative playing strength.

The empirical result is a final within-pool comparison. There is no reported controlled ablation isolating the causal benefit of tactical successors, opponent memory, branch exploration, entropy, or fan weighting. Their inclusion describes the system; it is not evidence that each component independently improves strength.

The branch target covers at most two games, although the policy is conditioned on the full match horizon. It can miss longer-term effects. Pairing controls the initial state and random plan but does not make continuations identical or establish a measured reduction in estimator variance; changes in action order alter which tiles are consumed.

Rare large-fan wins remain sparse even at this scale. Their counts should not be read as stable probabilities for every hand pattern. The weighted training objective favors selected outcomes and can differ from unweighted score maximization.

The released checkpoint is selected from evaluated candidates. There is no external human evaluation, independent benchmark, or multiple-training-seed estimate reported here. The original training dataset and comparison pool are not included, so exact end-to-end reproduction of the reported weights or 36-participant tournament is outside this release.

This report specifies a public-information Transformer agent, a simulator-based paired exploration procedure, and a frozen-pool evaluation for one Hangzhou Mahjong rule set. The released r2600 checkpoint has the highest observed score under the predefined tournament weighting, without a reliable leading-pair separation in the reported interval. The implementation and one checkpoint support new experiments; component effectiveness, training efficiency relative to alternatives, and generalization beyond this pool remain open empirical questions.

This README is the sole documentation file. Supporting simulator and policy modules are direct dependencies of training and evaluation.

README.md
requirements.txt
.gitattributes
.gitignore
evaluate.py                 # CPU/CUDA mirrored tournament entry point
reinforce.py                # r2600-based branch-training entry point
bench_pool.py               # mirrored table execution and policy interface
mj/                         # simulator, fan/legality/shape logic, public features
train/                      # model, features, supervision, branch RL, inference
weights/
  r2600.pt                  # selected policy checkpoint; no optimizer or dataset

The source tree does not include historical opponent policies, experiment archives, match logs, demonstration data, service credentials, deployment adapters, or other neural checkpoints. Hand-shape and unseen-tile calculations needed by the neural features remain as pure feature functions; they do not implement a rule opponent.

Checkpoint identity:

• Parameters: 52,005,062.
• File size: 208,168,496 bytes.
• File SHA-256: 6a5b4c29f830479b72b19b6e8ad4de33628b5331eb109a0fd99b026b67e9f0c2.
• Tensor-name/value digest: 467615daae168b787edcf7b05f857efa6250da302a2d56d1efe5350cbcb4a306.
• Contents: model configuration, parameter tensors, and public feature schema.
• No optimizer state is bundled; a new training run initializes a new optimizer.

The tensor digest is formed by iterating sorted parameter names, hashing each UTF-8 name and then its contiguous CPU NumPy bytes. The separate file digest also covers serialization metadata.

Use Python 3.11 or 3.12. The reference environment uses PyTorch 2.8. CPU evaluation requires PyTorch, NumPy, and Numba. The distributed branch trainer requires Linux, CUDA-capable PyTorch, Triton (provided with the supported Linux CUDA PyTorch installation), and SSH access to user-configured sampler hosts.

git lfs install
git clone https://github.com/MaxwellCCC/hangzhou-mahjong.git
cd hangzhou-mahjong
git lfs pull
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Choose a PyTorch build compatible with the intended CPU/CUDA environment. Run commands from the repository root.

The checkpoint is stored through Git LFS because it exceeds GitHub's normal single-file Git limit. Install Git LFS before cloning; in an existing checkout, run git lfs pull to fetch the checkpoint. The included .gitattributes configures weight tracking.

python evaluate.py \
  --device cpu \
  --self-play \
  --out runs/cpu_smoke

This runs r2600 in all four seats for a single-game table under four seat rotations: four actual games and sixteen seat-games for the same participant. Its aggregate net score is zero by construction. The report does not designate a champion or compute a pairwise confidence interval; this checks model loading and complete game execution, not strength.

A comparison against a user-supplied compatible checkpoint on a CUDA device:

python evaluate.py \
  --device cuda --devices 0 \
  --model candidate=/path/to/candidate.pt \
  --workers 4 --threads 2 \
  --seats 40000 \
  --out runs/comparison

Use --seats 120000 for three times the standard target. Repeat --model NAME=PATH to add more compatible checkpoints; comparative evaluation requires at least two distinct sets of weights, including r2600. Use --device cpu for a sequential CPU comparison. Each model seat receives its own history state. Resume an interrupted evaluation using the same arguments and output directory; a different frozen plan is rejected. Mirroring and balanced participation can round the requested seat count upward.

Outputs include plan.json, groups.jsonl, summary.json, and scores_and_fan.json. The summary records the weighted and pooled per-seat-game scores. The exact-fan report records raw cumulative scores, actual participation counts, and exact 8/16/32-fan wins. None of these generated run files belongs in the source release.

Install the same source tree and dependencies on the learner and sampler hosts. Configure SSH yourself. A minimal sampler JSON, saved outside the tracked source, is:

{
  "host": "your-sampler-ssh-alias",
  "project": "/path/to/this/repository",
  "python": "/path/to/venv/bin/python",
  "devices": [0],
  "prefetch_batches": 2,
  "prefetch_depth": 4,
  "continuous_prefetch": true,
  "share_prefix": true
}

For multiple sampler machines, use a "samplers" array containing per-host overrides, with shared settings at the top level. Model files are transferred to the sampler service by the training transport. Communication uses trusted SSH-connected processes and Python serialization; this is not an unauthenticated public inference API.

python reinforce.py \
  --samplers /path/to/samplers.json \
  --out runs/branch_rl \
  --rounds 100 \
  --update-devices 0

Use --update-devices 0,1 for the two-learner-GPU path. Resume a run with the same command and --resume. The round limit is the total target for that run, not an additional-round count.

The wrapper initializes the learner and one frozen neural opponent from the same r2600 checkpoint, with no bundled rule opponents. The internal name "router" in its generated opponent manifest is a role name for that frozen checkpoint, not a separate bundled router model. Training can add newly selected checkpoints to its own run's opponent pool. This is a new self-play population and optimizer state; it does not reconstruct the original training population or resume its old optimizer.

Advanced users can invoke python -m train.ppo_v4 --help to configure the training implementation directly.

The supervised loader supports two existing table formats:

TableDataset in train/action_inputs.py is the executable data contract. For the released tactical checkpoint, use a compatible tactical dataset with its per-decision "tactical" records; the basic NPZ loader alone does not synthesize these records.

A tactical dataset directory has a configuration such as:

{
  "schema": "v4_tactical_warmup",
  "tables": [
    {"path": "table_train.pt", "split": "train"},
    {"path": "table_validation.pt", "split": "validation"}
  ]
}

Those files must contain actual compatible table data; they are not supplied by this example. No corpus downloader or dataset-generation service is included.

python -m train.train_base_v4 \
  --data /path/to/tactical_dataset \
  --init weights/r2600.pt \
  --target router \
  --out runs/supervised \
  --lr 1e-4 \
  --epochs 7 \
  --checkpoint-layers

The initialization checkpoint defines the model architecture. Supervised continuation has separate learning-rate, auxiliary-loss, and validation settings from branch RL. It is not an automatic reproduction recipe for r2600.

1. Junjie Li et al. Suphx: Mastering Mahjong with Deep Reinforcement Learning. 2020. arXiv:2003.13590.
2. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. 2017. arXiv:1707.06347.
3. Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. 2019. arXiv:1910.07467.
4. Jianlin Su et al. RoFormer: Enhanced Transformer with Rotary Position Embedding. 2021. arXiv:2104.09864.
5. Noam Shazeer. GLU Variants Improve Transformer. 2020. arXiv:2002.05202.
6. Dimitri P. Bertsekas, John N. Tsitsiklis, and Cynara Wu. Rollout Algorithms for Combinatorial Optimization. Journal of Heuristics, 3:245–262, 1997. Author-hosted paper.
7. Andrew Y. Ng and Michael I. Jordan. PEGASUS: A Policy Search Method for Large MDPs and POMDPs. Proceedings of UAI, pp. 406–415, 2000. arXiv:1301.3878.
8. Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual Multi-Agent Policy Gradients. 2017. arXiv:1705.08926.

The project source code, this document, and the bundled r2600 checkpoint are provided under the MIT License. External dependencies retain their own licenses.

Copyright (c) 2026 MaxwellCCC

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.