Random Number Generation Requirements

Intent

This is a collaborative document for developers to work in to build up a common understanding of what we want to achieve in terms of offering a reasonably good RNG design with a consistent implementation so that users can manage their RNG appropriately for their particular study.

At some point, this document will have served its purpose and we will likely neither finish nor maintain it. We will endeavor to update this section when that occurs.

User Stories

TODO: Add in information about multiple sampler usage in single application. Use cases or user stories?

As a user, I would like

1. documentation that clearly describes how RNG is implemented in surmise so that I can assess if it is sufficient for my research, understand what my responsibilities in managing random number generation are, and setup/execute my studies responsibly.

2. RNGs to be high enough quality that they support the current community standards for use of RNGs in statistics-based research. In particular, use of RNGs that adhere to the strictest random number generation standards for security and cryptography is welcome but not required.

3. to be able to control the random number generation in surmise using only one random number generation system so that the use of RNGs in surmise is simple and easy to understand/use as well as to be able to understand how its use of RNGs fits with any random number generation in my code.

4. to be able to create a set of statistically independent RNGs from that single RNG system so that I can run a set of independent trials if needed.

5. to be able to use an RNG configured not only by seed, but also by my explicit choice of RNG method (e.g., choose SFC64 for speed or PCG-64 DXSM for improved statistical results when using parallelized codes).

6. to be able to fix RNGs in my deterministic applications so that I can confirm basic reproducibility of all numerical results under the same software conditions.

7. to be able to access the MCMC samples in the order in which they were obtained during calibration in order to analyze the calibration process and study the convergence to the target posterior distribution.

8. to have a convenience function that provides the MCMC samples with a randomized reordering so that these act as a simulation of sampling from a known distribution as part of a normal Monte Carlo integral estimation process. This could be useful, for instance, for simulating the convergence properties of different integrated results as the number of MC samples increases.

As a developer/maintainer, I would like

1. to provide an interface that allows users to control random number generation as needed for their studies. While surmise developers must put forth their best effort to use RNGs appropriately in the package and communicate their use clearly to users, we shall make it clear that users are ultimately responsible for the overall use of RNGs in their study.

2. to restrict random number generation throughout to the consistent use of a single random number generation system so that the design/implementation is clean and developers only need to understand one system well. This should also aid maintenance since maintainers would only have to deal with adjusting surmise to interface changes in only one system.

3. to use a single random number generation system that also contains other typical statistical software tools needed for surmise so that the design/implementation is simpler/cleaner and statistical software tools are interrelated (e.g., same definitions), co-developed, and encapsulated.

4. to be able to fix RNGs in tests and examples so that all numerical results obtained with deterministic codes are reproducible under the same software conditions. We should be able to add tests of identically reproducible results so that these can be used effectively for code refactoring.

5. to be able to design and implement units of computation (e.g., a single calibration or single emulation) that require the use of a single RNG instance so that users cannot accidentally change the RNG during the middle of that unit's computation.

6. to determine for each calibration a single randomized reordering of its MCMC samples as part of calibration so that the use of the RNG occurs entirely during calibration rather than potentially splitting its use across calibration (online) and post-calibration analysis (offline). If the surmise interface provides access to this random reordering of the MCMC samples, ideally it would always return this single reordering as THE results of simulating a normal Monte Carlo integral estimation process using MCMC techniques, which removes the need of surmise to use RNGs for post-calibration work.

7. to potentially re-use random numbers (e.g., through common random numbers) as a variance reduction technique.

8. to design RNG use in the MCMC samplers so that they are more strongly decoupled from surmise so that their implementation is potentially more general and, therefore, could be used outside the package or for additional tasks. In particular, it is acceptable if calibrators, which exist in surmise at a higher, more public level,

• access an RNG as detailed above,
• determine RNG usage for the broader calibration task, and
• pass the necessary RNG to the sampler as an argument.

In particular, samplers will not use the surmise RNG system directly nor know/care that it exists.