Skip to content

[ENG-08] RNG handling is inconsistent; reproducibility is a vibe, not a contract #290

@kgdunn

Description

@kgdunn

Tracked as ENG-08 in ENGINEERING_AUDIT.md.

Severity: Major

Problem

Unseeded RNGs in production code:

  • process_improve/experiments/simulations.py:101 -- np.random.default_rng().normal(0, 1).
  • process_improve/simulation/model.py:425 -- noise_rng = np.random.default_rng() (documented as "fresh noise on every call"; arguably intentional).
  • process_improve/multivariate/methods.py:5770, :5794 -- Resampler.bootstrap / .fractional (also tracked as SEC-21 sub-item 9, [SEC-21] NaN-poisoning in single-block PCA / PLS / TPLS #270).

Hard-coded seeds in production code:

  • process_improve/experiments/optimization.py:564 -- rng = np.random.default_rng(42).
  • process_improve/experiments/visualization/plots/surfaces.py:242 -- seed=0.
  • process_improve/experiments/visualization/plots/design_quality.py:85 -- seed=42.

The @tool_spec(rng={...}) metadata is well-intentioned but enforces nothing -- the decorator only validates the shape of the metadata, not that the function obeys it.

Fix direction

Publish a one-page reproducibility contract (docs/reproducibility.rst):

  1. Every public function that touches an RNG MUST accept random_state: int | np.random.Generator | None.
  2. Hard-coded seeds in production code are forbidden (move defaults into function signatures).
  3. The rng={} decorator should unit-test itself: a test that imports every registered tool, builds a random input, runs it twice with the declared seed, and asserts the outputs match.

Acceptance

  • The contract is documented.
  • No unseeded np.random.default_rng() in production code (except where explicitly justified).
  • A tests/test_rng_contract.py enforces (3).

Cross-references

SEC-21 sub-item 9 (#270) and SEC-33 sub-item on optimization.py:564 (#282) overlap; this issue covers the policy layer they imply.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions