Tests: Strange flakiness in TestMarginalVsLatent GP test on Windows

[maresb]

Description

I noticed this. Probably very low-priority, but just wanted to document it:

run: windows (windows-latest, float64, 3.11, tests/gp/test_cov.py tests/gp/test_gp.py tests/gp/test_me...
link:
https://github.com/pymc-devs/pymc/actions/runs/20239861021/job/58104657875

Since the run logs get deleted eventually, I'm attaching the relevant output here

 _________________ TestMarginalVsLatent.testLatentMultioutput __________________

self = <tests.gp.test_gp.TestMarginalVsLatent object at 0x00000268829F54D0>

    def testLatentMultioutput(self):
        n_outputs = 2
        X = np.random.randn(20, 3)
        y = np.random.randn(n_outputs, 20)
        Xnew = np.random.randn(30, 3)
        pnew = np.random.randn(n_outputs, 30)
    
        with pm.Model() as latent_model:
            cov_func = pm.gp.cov.ExpQuad(3, [0.1, 0.2, 0.3])
            mean_func = pm.gp.mean.Constant(0.5)
            latent_gp = pm.gp.Latent(mean_func=mean_func, cov_func=cov_func)
            latent_f = latent_gp.prior("f", X, n_outputs=n_outputs, reparameterize=True)
            latent_p = latent_gp.conditional("p", Xnew)
    
        with pm.Model() as marginal_model:
            cov_func = pm.gp.cov.ExpQuad(3, [0.1, 0.2, 0.3])
            mean_func = pm.gp.mean.Constant(0.5)
            marginal_gp = pm.gp.Marginal(mean_func=mean_func, cov_func=cov_func)
            marginal_f = marginal_gp.marginal_likelihood("f", X, y, sigma=0.0)
            marginal_p = marginal_gp.conditional("p", Xnew)
    
        assert tuple(latent_f.shape.eval()) == tuple(marginal_f.shape.eval()) == y.shape
        assert tuple(latent_p.shape.eval()) == tuple(marginal_p.shape.eval()) == pnew.shape
    
        chol = np.linalg.cholesky(cov_func(X).eval())
        v = np.linalg.solve(chol, (y - 0.5).T)
        A = np.linalg.solve(chol, cov_func(X, Xnew).eval()).T
        mu_cond = mean_func(Xnew).eval() + (A @ v).T
        cov_cond = cov_func(Xnew, Xnew).eval() - A @ A.T
    
        with pm.Model() as numpy_model:
            numpy_p = pm.MvNormal.dist(mu=pt.as_tensor(mu_cond), cov=pt.as_tensor(cov_cond))
    
        latent_rv_logp = pm.logp(latent_p, pnew)
        marginal_rv_logp = pm.logp(marginal_p, pnew)
        numpy_rv_logp = pm.logp(numpy_p, pnew)
    
        assert (
            latent_rv_logp.shape.eval()
            == marginal_rv_logp.shape.eval()
            == numpy_rv_logp.shape.eval()
        )
    
>       npt.assert_allclose(latent_rv_logp.eval({"f": y}), marginal_rv_logp.eval(), rtol=1e-4)
E       AssertionError: 
E       Not equal to tolerance rtol=0.0001, atol=0
E       
E       Mismatched elements: 2 / 2 (100%)
E       Max absolute difference among violations: 4.21352582
E       Max relative difference among violations: 0.00120481
E        ACTUAL: array([-3501.471817, -2764.690228])
E        DESIRED: array([-3497.258291, -2761.368021])

tests\gp\test_gp.py:334: AssertionError

[Jiya873]

Hi, I’d like to work on this.

This looks like a Windows-specific numerical precision issue (likely due to linear algebra backend differences). I plan to slightly relax the rtol in testLatentMultioutput to avoid these flaky failures.

[maresb]

Great @Jiya873!

I noticed that it doesn't seem to fail consistently among all the runs, so I suspect that some non-determinism might be at play. You might alternatively look into setting a random number generator seed.

It might not even be Windows-specific.

Although not strictly required, it would be interesting if you could find a way to reproduce it locally.

[Jiya873]

That makes sense, I’ll first look into making the test more deterministic by setting an explicit random seed and see if that resolves the flakiness. I’ll also try to reproduce the failure locally if possible.

If the issue persists after that, I can then consider relaxing the tolerance as a fallback. I’ll report back with what I find.