Seed HSGP bootstrap tests #6679
Conversation
ricardoV94
added
tests
needs info
ricardoV94
force-pushed
the
seed_hsgp_bootstrap
branch
from
fe336b9
to
48c4b25
Compare
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #6679 +/- ##
==========================================
- Coverage 91.99% 89.52% -2.47%
==========================================
Files 94 94
Lines 15944 15944
==========================================
- Hits 14667 14274 -393
- Misses 1277 1670 +393 |
|
weird that a different seed it still fails. maybe very unlucky...? I couldnt get it to fail in the original PR. Does it fail for like, any seed? |
|
My last commit tries 5 seeds, it fails in one of them. The failure rate seems higher than 0.01. Even with 5 tests should only fail 5% of the time |
|
yup somethings wrong, will pull and give it a try |
| seeds = np.arange(5) + 197 # 5 possible seeds | ||
| rng = np.random.default_rng(np.random.choice(seeds)) | ||
|
|
There was a problem hiding this comment.
It seems cleaner to use @parameterize like for test_prior?
There was a problem hiding this comment.
The @parametrize was just to show what seed caused the test to fail. The idea was to revert back to choosing one of 5 possible seeds once the root cause was found.
I imagine we don't want to run each test 5 times, but instead allow some variation across tests.
| """Compare HSGP prior to unapproximated GP prior, pm.gp.Latent. Draw samples from the | ||
| prior and compare them using MMD two sample test. Tests both centered and non-centered | ||
| parameterizations. | ||
| """ | ||
| rng = np.random.default_rng(seed) | ||
|
|
||
| with model: | ||
| hsgp = pm.gp.HSGP(m=[200], c=2.0, parameterization=parameterization, cov_func=cov_func) |
There was a problem hiding this comment.
try increasing m here to 500
| with model: | ||
| hsgp = pm.gp.HSGP(m=[200], c=2.0, parameterization=parameterization, cov_func=cov_func) | ||
| f1 = hsgp.prior("f1", X=X1) | ||
|
|
||
| gp = pm.gp.Latent(cov_func=cov_func) | ||
| f2 = gp.prior("f2", X=X1) | ||
|
|
||
| idata = pm.sample_prior_predictive(samples=1000) | ||
| idata = pm.sample_prior_predictive(samples=1000, random_seed=rng) |
There was a problem hiding this comment.
I think its ok to decrease samples here to 500, will run faster which will be nice for trying the different random seeds
| @@ -194,17 +197,20 @@ def test_conditional(self, model, cov_func, X1, parameterization): | |||
| prior and compare them using MMD two sample test. Tests both centered and non-centered | |||
| parameterizations. The conditional should match the prior when no data is observed. | |||
| """ | |||
| seeds = np.arange(5) + 197 # 5 possible seeds | |||
| rng = np.random.default_rng(np.random.choice(seeds)) | |||
|
|
|||
| with model: | |||
| hsgp = pm.gp.HSGP(m=[100], c=2.0, parameterization=parameterization, cov_func=cov_func) | |||
There was a problem hiding this comment.
try increasing m here to 500 too. This got rid of the random failures for me locally
Failed in #6678
📚 Documentation preview 📚: https://pymc--6679.org.readthedocs.build/en/6679/