primegap

MDM-Prime represents each token as a length-ell string of base-b sub-tokens and masks the sub-tokens independently, so a token can be partially masked. Its decoder models the joint distribution of a token's ell sub-tokens, and the training objective is the masked-diffusion loss with that joint reconstruction term:

L_vb = integral_0^1  (alpha'_t / (1 - alpha_t))  E[ sum_i log p_theta(y_0^i | y_t) ] dt,

where the sum runs over token positions and each p_theta(y_0^i | y_t) is the joint over that token's sub-tokens. The paper states this is a variational upper bound on the negative log-likelihood, L_vb >= -log p_theta(x_0).

It is not an upper bound. At the Bayes-optimal decoder it falls below the data entropy whenever a token's base-b sub-tokens are statistically dependent, which is the case for every real token distribution.

The bound falls below the entropy

Any valid variational bound satisfies E[L_vb] >= H by Gibbs' inequality: the expected negative log-likelihood of any normalized model is at least the data entropy H. Evaluating L_vb at the optimal joint decoder (the true posterior of the masked sub-tokens given the unmasked ones, with unmasked sub-tokens pinned by carry-over) gives a Beta-weighted sum of conditional entropies,

L_vb = sum_{S subsetneq [ell]}  ( |S|! (ell - |S| - 1)! / ell! )  H( masked | S ),

which for ell = 2 is

L_vb = (3/2) H - (1/2)( H(y^1) + H(y^2) )  =  H - (1/2) I(y^1 ; y^2).

So H - L_vb = (1/2) I(y^1 ; y^2) >= 0, with equality only when the two digits are independent. The objective sits below the entropy by half the multi-information of the digit code.

The controls fix the sign and the weighting: an atomic token (ell = 1) and a token with independent digits both give L_vb = H exactly, so in those cases the objective is the genuine masked-diffusion NELBO. Only sub-token dependence pushes it below H.

case	L_vb vs H
ell = 1 atomic token	L_vb = H (valid, tight)
ell = 2 independent digits	L_vb = H (valid, tight)
ell = 2 correlated [[.45,.05],[.05,.45]]	L_vb = 0.834 < H = 1.018, gap = I/2
ell = 3 correlated	L_vb < H

Every real token distribution violates it

The base-b encoding assigns arbitrary digit strings to token ids, and a real token law is highly non-uniform, so the induced digit distribution is dependent and the multi-information is positive. A Zipf token law and a random skewed law both drive L_vb strictly below H; only a distribution engineered so the digits are independent (a product law, never the case for language or code) is safe. Because L_vb can drop below the entropy, the perplexities it reports are not valid upper bounds and can fall below the true entropy, which is consistent with MDM-Prime reporting perplexity better than an autoregressive model of the same data.

Scope

The masked-diffusion machinery is otherwise intact: at ell = 1 this is exactly the standard MDLM bound, and the schedule and sampler are unaffected. What is not true is the stated inequality L_vb >= -log p_theta(x_0). The single, load-bearing step that fails is applying the per-unit MDLM weight to a joint-over-sub-tokens reconstruction term while masking the sub-tokens independently: that combination silently discards the within-token multi-information and lands below the entropy.

Layout

• elbo.py: L_vb at the optimal joint decoder for one token, by both a time integral and the exact Beta-weighted conditional-entropy sum.
• gap.py: the multi-information, the H - L_vb gap, and the ell = 1 and independent-digit controls that reproduce H exactly.
• tokenizer.py: folding a token law into its base-b digit law, and Zipf / skewed cases that all violate the bound.
• test_primegap.py: the controls, the correlated break, the exact I/2 gap, the closed form matching the integral, the ell = 3 violation, and Zipf violating while a product law does not.

Reproduce

python elbo.py
python gap.py
python tokenizer.py
python test_primegap.py

Everything runs in float64. The ell = 1 and independent-digit controls reproduce H to 1e-7; the closed form matches the integral to 1e-3; the ell = 2 gap equals I/2 to the integration tolerance.