negvar

pytorch_optimizer's QHAdam keeps an exponential moving average of the squared gradient as its second raw moment and divides the step by its square root. The second-moment line drops a pair of parentheses: it adds 1.0 - beta2_adj * grad_p2 instead of (1.0 - beta2_adj) * grad_p2. The missing parentheses inject a constant 1.0 and attach a minus sign to beta2_adj * grad_p2, so the accumulator is no longer a moving average of a non-negative quantity. For gradient magnitudes above one it is driven negative, and its square root is NaN within a few steps; for magnitudes below one it inflates the denominator and silently corrupts the trajectory. The adjacent first-moment line is parenthesized correctly, so the second-moment form is a typo.

The code

pytorch_optimizer, pytorch_optimizer/optimizer/qhadam.py, QHAdam.step, 3.10.1 and current main (byte-identical):

beta2_adj = 1.0 - (1.0 / beta2_weight)                       # in [0, 1)
grad_p2 = grad.pow(2)

exp_avg.mul_(beta1_adj).add_((1.0 - beta1_adj) * grad)       # line 128: parenthesized, correct
exp_avg_sq.mul_(beta2_adj).add_(1.0 - beta2_adj * grad_p2)   # line 129: parentheses missing

avg_grad_rms = exp_avg_sq.mul(nu2)                           # nu2 defaults to 1.0
avg_grad_rms.sqrt_().add_(group['eps'])                      # sqrt of the accumulator
p.addcdiv_(avg_grad, avg_grad_rms, value=-group['lr'])

Python parses line 129 as beta2_adj * exp_avg_sq + (1.0 - (beta2_adj * grad_p2)), not the intended beta2_adj * exp_avg_sq + (1.0 - beta2_adj) * grad_p2. The per-step difference from the correct update is exactly 1.0 - grad_p2, so the two agree when grad_p2 == 1, the accumulator inflates when grad_p2 < 1, and it goes negative when grad_p2 > 1.

What it does

Driving the real optimizer against a subclass with only the parentheses fixed (fp64):

|g|=2: second raw moment (EMA of g^2, must be >= 0) = -1.998  -> param is NaN: True
|g|=1: real -0.010000 vs fixed -0.010000  match: True
|g|=0.5: real exp_avg_sq 11.8188 vs correct 0.2500 (g^2=0.25); param real -0.00686 vs fixed -0.03000

The second raw moment is an EMA of the squared gradient and so must be non-negative, but the real QHAdam drives it to -1.998 after three steps of |g|=2, and the square root of the negative accumulator makes the parameter NaN on default hyperparameters. The |g|=1 case is the base-case control: because the per-step error 1 - g^2 is zero there, the buggy and fixed optimizers agree to machine precision, which isolates the single line as the fault and confirms the fixed reference is faithful. Below one the accumulator does not go negative but inflates: at |g|=0.5 the real exp_avg_sq reaches 11.82 where the correct value is 0.25, a roughly fortyfold denominator that silently shrinks and distorts the step with no crash.

Scope

QHAdam triggers this on default hyperparameters (betas=(0.9, 0.999), nus=(1, 1)) and common parameter shapes: any real training run produces gradient magnitudes above roughly 1.7 at some point, so the second moment goes negative and the run NaNs; runs that stay below one train on a silently wrong denominator. The docstring describes standard QHAdam with no note of this behaviour, and the first-moment line is parenthesized correctly, so the fix is to parenthesize the second-moment line the same way.

Layout

• excerpt.py: the correct first-moment line, the missing-parentheses second-moment line, and the square root that consumes the accumulator, quoted with the flags that name the fault (Apache-2.0).
• qhadam.py: the correct and buggy second-moment recurrences under the optimizer's own beta2_adj schedule, so they agree at g^2 = 1, inflate below one, and go negative above one.
• consequence.py: the real QHAdam versus a one-line-fixed subclass, the negative second moment and NaN, the |g|=1 agreement, and the |g|=0.5 silent inflation.
• test_negvar.py: the model agrees at g^2 = 1, stays non-negative when correct but goes negative when buggy above one and inflates below one; and the real optimizer's second moment goes negative to NaN, agrees at |g|=1, and inflates the denominator at |g|=0.5.

Reproduce

python qhadam.py
python consequence.py
python test_negvar.py

The two moment updates are quoted from current main; the numbers are produced by the real optimizer. The fix is to parenthesize the second-moment line as (1.0 - beta2_adj) * grad_p2, so the accumulator stays a non-negative moving average of the squared gradient.