Add SPEC7: Seeding pseudo-random number generation

[stefanv]

Under discussion at scipy/scipy#14322

[tupui]

That's a good idea to try to uniformize all that 👍

(I suppose you meant SPEC and not NEP.)

[tupui]

This is describing the current state in some libraries. But is it where we want to see this 10 years from now?

I am personally against any global state and advertising of any "legacy" behaviours.

[stefanv]

I also feel we may want to think about a new keyword argument instead, that adopts recommended best practices instead.

[tupui]

I am not sure I see how that could help.

If you add a rng to a function which has seed or random_state we don't avoid raising some warning about deprecation, etc.

[stefanv]

The big problem with random_state is that it allows for None, which then grabs global state. So, that will always conflict with an rng=None kwarg.

[tupui]

To me, the ideal API (using todays tool at least) would be that random_state=None would give you np.random.default_rng().

There is also the crazy thought, which I kind of like, from @ilayn: do not accept integers, only a Generator (or other object). Point being, you must provide a RNG if you want any reproducibility.

[stefanv]

I don't think we can do that, though, because it would be a backward incompatible change?

[stefanv]

Accepting only Generator objects could work, but we still have to deal with None.

[tupui]

Both integer and global state behaviour are BC issues yep.

It would be interesting to see how in real life Generator was painful to move to, taking into account a large sample of projects, folks, etc.

i.e. sometimes I feel like we are concerning ourselves too much about BC while for users it might be super easy and accepted to make the change. It's mostly a communication issue to me. Always taking my backend example, but there they do break (intentionally or not) production code like all the time. There are complains yep, but it's mostly ok and the ball is rolling and these projects are loved and praised (FastAPI is known for doing this often and seen as THE thing.)

[stefanv]

Yes, and don't expect the same favors from commercial code. It's shackles that we choose to put on, sometimes at great time cost to (oftentimes volunteer) developers.

[rkern]

The big problem with random_state is that it allows for None, which then grabs global state. So, that will always conflict with an rng=None kwarg.

There's a deprecation strategy that can work to migrate from random_state to rng, if one wants that. Functions will (for a time) take random_state and rng arguments. Have a check_rng(rng=None, random_state=None) function that will return a Generator given its arguments that function authors can use. You can change the behavior over time, with DeprecationWarnings. So on the first release, if rng=None, then it looks at random_state and start issuing DeprecationWarnings, but otherwise using the same semantics as check_random_state(random_state) (but taking out the BitGenerator of the resulting RandomState and wrapping it in a Generator instead). Then when you enforce the deprecations, you can migrate to just returning default_rng(rng) and start raise informative errors if something other than None is passed to random_state, then eventually you can drop the random_state= argument entirely.

[stefanv]

There's a deprecation strategy that can work to migrate from random_state to rng

To make sure I understand, this will change the return values of functions (rng=None, random_seed=None) over time (i.e., violate the Hinsen principle), but we can choose how long that time period is.

[rkern]

Yes, it's a deprecation strategy, not a backwards-compatibility-preserving strategy.

[rgommers]

@stefanv thanks for starting to summarize that long and complex discussion!

Specifically, behavior will change over a long period of time in the case where no seed is specified

Can you please elaborate on this? It's not all that obvious, because when you're not seeding the first intuition I'd have is "I am not expecting specific results, only random numbers with a given distribution". Since you're kinda steering towards a large amount of churn due to changing names here, I think it's important to be specific under what circumstances there is a backwards compatibility impact.

I guess the point here is:

• the user may be seeding elsewhere with np.random.seed(a_number),
• and not threading the state for that seed through to this API call,
• and executing without multiprocessing or a similar parallel mechanism,
• hence was relying implicitly on the global state controlling by seeding,
• this global state is now no longer determining the exact random numbers to the unseeded API call,
• hence the numerical result returned by that API call changes,

And then there's the question whether this scenarios matter. It may impact exact reproducibility of some scientific result. However that reproducibility was only ever guaranteed when using the same version of the same libraries on the same hardware.

I'd suggest finding the most compelling scenario here, that makes it as easy as possible to say that that's not acceptable, and hence we must change from random_state to rng.

[rkern]

The deprecation strategy I outlined does imply a change in semantics of the affected functions above and beyond the change in the precise numbers that come out of them. There are plenty of programs (using scipy and sklearn components that use check_random_state()) that rely on controlling the output deterministically (for the same program, same builds, same environment, not across versions or builds) by calling np.random.seed(seed) once at the top rather than explicitly threading through a RandomState. Following that deprecation, those programs won't be able to be made deterministic anymore without some (small) rewriting.

[stefanv]

Following that deprecation, those programs won't be able to be made deterministic anymore without some (small) rewriting.

Right, and my take was that this is a desired outcome from our perspective.

[stefanv]

And then there's the question whether this scenarios matter.

By far the most common use of seeding is to fix test suites. Most of those will keep running as-is. The failures that arise will be legitimate failures, and could be fixed by playing with the seed, or by making the underlying code more robust.

[ilayn]

I am willing to spend time on the rewriting. The test suite is seriously out-of-date in many places anyways. You can even smell the year from just by reading the comments.

[stefanv]

I've made more explicit the points you mentioned, Ralf. It may benefit from fleshing out even further as we continue to evolve the document. I don't want to tighten things up before we've agreed on a pathway forward!

[rkern]

Right, and my take was that this is a desired outcome from our perspective.

Yes, I think so. But I interpreted Ralf's question as whether it was really necessary to go through a deprecation and a name churn to do this instead of just changing what random_state=None does since stream reproducibility across versions is not something that most of the downstream libraries using the check_random_state() pattern guarantee. I was confirming that there was something beyond the stream reproducibility that is a concern.

[rgommers]

rely on controlling the output deterministically (for the same program, same builds, same environment, not across versions or builds) by calling np.random.seed(seed) once at the top rather than explicitly threading through a RandomState. Following that deprecation, those programs won't be able to be made deterministic anymore without some (small) rewriting.

Yes, I think that's saying exactly the same thing I was saying in my bullet points higher up. I would add that library code doing this is already broken, because it's not robust to (for example) the end user using multiprocessing. So I think what you're worried about is end user code doing this. And missing the change in semantics, then re-run experiments after upgrading to the next scipy version and saving results that they think are deterministic but silently aren't. Right?

[stefanv]

Why don't we emit a warning from numpy random.seed?

[rgommers]

Because it will create utter havoc in the many valid uses in test suites?

[stefanv]

Isn't that what you want, eventually?

[stefanv]

Phrased differently: once we deprecate global seeding for the ecosystem, what would be the use of np.random.seed?

[rkern]

So I think what you're worried about is end user code doing this. And missing the change in semantics, then re-run experiments after upgrading to the next scipy version and saving results that they think are deterministic but silently aren't.

Yes. There are plenty of ML programs (in particular) that call np.random.seed(seed) (and random.seed(seed) and torch.manual_seed(seed), etc.) at the top because that's what they've been told to do (and encouraged to do by well-meaning frameworks) and call sklearn and scipy functions, and those scripts are deterministic (and mostly fine). That will silently change if we just change check_random_state(None) to do default_rng(None) without a deprecation switcheroo. I care less about the results they get from any one run (they'll be perfectly valid) so much as they will now show unexpected and hard-to-debug behavior as everyone chases down a bunch of red herrings.

Phrased differently: once we deprecate global seeding for the ecosystem, what would be the use of np.random.seed?

One enormous hurdle at a time, please. 😉

[stefanv]

One enormous hurdle at a time, please. 😉

Fair enough :)

[rgommers]

Phrased differently: once we deprecate global seeding for the ecosystem, what would be the use of np.random.seed?

There is no plan to do so. Deprecating random_state=None in scipy and (hopefully after that) scikit-learn is a very different thing from deprecating the legacy numpy.random functionality. There is no such plan, it's legacy only and NEP 19 explicitlly lays out that it will stay, be ultra-stable, and useful for testing.

[rgommers]

and those scripts are deterministic (and mostly fine)

Yes, okay - I agree, this summary and rationale is enough to explain why we cannot stay with random_state, I think the average reader will understand this well enough.

That also means that item (a) of my reasoning in scipy/scipy#14322 (comment) is not "in the same ballpark" and hence it seems clear now that we should prefer rng over random_state.

[tupui]

There is no such plan, it's legacy only and NEP 19 explicitlly lays out that it will stay, be ultra-stable, and useful for testing.

I think that would not prevent from having a user warning. Average users don't read docs and keep copy pasting old code until "something" is getting in their ways. So until it's visible in their code that something is legacy they will keep using that I am afraid.

Also reading at the NEP19, to me it's really not clear that the global state would not change. The fate of RandomState is clear, but the rest of the global section even ends with:

This NEP does not propose that these requirements remain in perpetuity. After we have experience with the new PRNG subsystem, we can and should revisit these issues in future NEPs.

[stefanv]

There seems to be some vague consensus around the deprecation approach. I don't want to run things ahead, but at the same time scikit-image has to make a calculated guess of what to do for its forthcoming release. So, without holding anyone to the fire, I will propose that we make the seed (random_state) to rng transition there, and hope that it doesn't cause too much work in the future should the decision here go differently.

I would appreciate it if those involved in the discussion would co-author this SPEC (whether by adding your name to the authors list, or by helping to clarify language). If you want to keep a safe distance, advice on how to solidify the thrust of the argument further would also be welcome.

Thanks!

[stefanv]

skimage completed the transition for our 0.21 release (scikit-image/scikit-image#6922). Maybe premature, we'll see. But at least usage is consistent now.

[bsipocz]

I wonder whether this converged enough to be merged as a draft, certainly has more content and usage than some of the other specs?

[stefanv]

Let me chat to the NetworkX people at their community call next week, and see what their thoughts are. While this is now implemented in scikit-image, it would be good to have at least multi-project consensus on this being a viable route forward.

[jorisvandenbossche]

Comment from the sideline (and as library author who recently added a first method to geopandas where we needed to decide which keyword name to use for this, without being constrained by existing methods):
From a user API perspective, I find the keyword name "rng" rather unclear / unintuitive. You already need to know it is short for "random number generator" (I assume), but I think many users won't know that. In contrast, something like "random_state" is much clearer from the name that it has to do with the random aspect of the function.

[stefanv]

As you point out, if you start from scratch, using random_state is an option. But many libraries already use random_state=None to refer to the old RandomState. They cannot easily keep using that keyword without breaking backward compatibility in an unexpected way.

So, the question is not so much finding an optimal argument name for one library, but coming to consensus over a consistent pattern that can be adopted across the ecosystem.

[jorisvandenbossche]

Yes, I fully understand that the deprecation process limits what can be used, but I hope someone could come up with a better name than "rng" and which is not "random_state" .. (although I don't directly have an idea myself, except from spelling it out, which would make it rather long)

[rkern]

FWIW, I responded to a similar comment on the related scipy thread.

[tupui]

I agree that ideally we would want to have a descriptive name for non experts and newcomers.

Has the ship really sailed to still use random_state but change the semantic progressively? (there are a few options like using an env variable) Because at some point, whatever we do, it seems like we are going to introduce some churn.

Otherwise I would be +1 on trying to brainstorm a new name.

randomness, chaos_state, etc. I am sure we can come up with something 😃

[stefanv]

Environment variables are never an option: it has to be explicit in the code.

[rkern]

Yes, that ship is long gone over the horizon.

I think the other ship is gone, too. With default_rng() and all of the numpy documentation using rng = default_rng(), rng is here to stay as a term that everyone will become familiar with.

TBF, I wanted prng = default_prng() to emphasize the pseudo aspect of the pseudorandomness, but people wanted something even shorter than that.

[stefanv]

FWIW, I don't think rng will be hard to teach: users will read it once and get it, especially if we make it a common pattern across the ecosystem:

rng = np.random.default_rng(x)

Rng, the thing we want as input, is already enshrined in the NumPy function name too.

prng would have been slightly better from an educational point of view.

random_generator is a longer form that is quite intuitive and also captures the notion of a seed adequately.

[stefanv]

Ah, sorry Robert, didn't see your message there.

[betatim]

For what it is worth, I think rng is no more or less arbitrary (some might say random ..) than random_state. You gotta learn it once and then you know it. Consistency across the ecosystem will be super useful here (just like it is for random_state).

I think rng has a nice ... ring to it ;) (I couldn't resist)

[tupui]

I went over the current document and strategy. I am +1.

I don't like the name 😅 but yes this is something users will eventually get with the warnings; and being written all over in NumPy's doc is helping us to sell it.

[asmeurer]

The array API discussion about random number generation and the differences between the NumPy-style and JAX-style APIs may or may not be relevant here data-apis/array-api#431. I don't know if it matters for seeding specifically, but I also know a lot of the same people on that discussion are already on this one.

[EwoutH]

Thanks for this effort! It might be useful to add a Motivation and/or Context section to this spec. I think it makes it a bit clearer why this spec was created and where it came from.