Skip to content

Every CustomSampler in a stochastic model is reset with the same seed #1093

Description

@thc1006

Describe the bug

Every CustomSampler attached to one stochastic model is reset with the same seed. StochasticModel._validate_custom_sampler ends in

sampler.reset_seed(seed)

and seed is the model's seed, passed unchanged to each sampler in turn. A sampler that follows the documented pattern and builds np.random.default_rng(seed) therefore starts from the same state as every other sampler in that model. Two parameters that share a distribution family draw the same value, and two that do not still draw the same underlying deviate.

Quoting the NumPy documentation on parallel random number generation:

NumPy allows you to spawn new (with very high probability) independent BitGenerator and Generator instances via their spawn() method.

and, in the section on sequences of integer seeds, on deriving several seeds from one root:

It is quite likely that multiple invocations of the program with different seeds will get overlapping sets of worker seeds [...] subsets of the workers will return identical results, causing a bias in the overall ensemble of results.

Handing one integer to several independent generators is the case that guidance exists to steer away from. Here it is not merely likely to overlap, it is the same integer every time.

To Reproduce

Two samplers, different means and different standard deviations, on one rocket:

class Gaussian(CustomSampler):
    def __init__(self, mean, sd):
        self.mean, self.sd = mean, sd

    def sample(self, n_samples=1):
        return list(self.rng.normal(self.mean, self.sd, n_samples))

    def reset_seed(self, seed=None):
        self.rng = np.random.default_rng(seed)


rocket = StochasticRocket(
    rocket=calisto, mass=Gaussian(14.426, 0.5), radius=Gaussian(0.0635, 0.001)
)
rocket._set_stochastic(4242)
drawn = next(rocket.dict_generator())

Recovering the standard normal deviate behind each draw:

mass     14.659110385288670   ->  0.466220770577340
radius   0.063966220770577    ->  0.466220770577341

The two agree to fifteen significant figures. The last digit is the scaling, not the draw. Every later pair agrees the same way, so the correlation is exactly 1 rather than merely high, and a study that varies both parameters is really varying one.

Expected behavior

Separate samplers are separate parameters and should be independent, with the model's seed deciding all of them together. Spawning one SeedSequence child per sampler would do that and stay reproducible.

The obvious fix has a case to answer first. docs/user/custom_sampler.rst documents a pair of wrappers that share one bivariate generator on purpose, so that wind X and wind Y stay correlated. Giving every sampler its own child would break that pattern, so the fix needs a way to say "these two share a stream deliberately" rather than simply spawning per attribute.

Additional context

Not a regression. This is the behaviour on develop and predates #1054, which changes neither _validate_custom_sampler nor its call sites. Found while reviewing that PR.

Related: #1053 (Monte Carlo reproducibility), #1042 (sensor noise outside the seed tree).

Verified on develop at 1691119, NumPy 2.5.1, Python 3.12.13.

Signed-off-by: thc1006 84045975+thc1006@users.noreply.github.com

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions