Add a feature flag to OsRng to allow use of RDRAND on modern systems

[raphaelcohn]

This is related to pull request #109. I've opened this as a new issue as I think this is something that needs proper tracking.

When working with highly concurrent modern data structures, it's often important to be able to pick a (hyper) thread local random number very, very quickly for a small value - a u32 or u64 or usize, for example. The RDRAND instruction on Ivy Bridge and later CPUs makes this straightforward, but one then needs a fallback - and the rand crate provides ThreadRng. This, however, eats into thread TLS space (threads aren't necessarily present on low-end platforms), requires initialisation, etc.

It'd be far more elegant to have perhaps a create compilation feature, or even better, when #[cfg(target_feature)] lands, to be able to have the OsRng switch over to RDRAND instead of using a syscall, and then to be able to eliminate ThreadRng set up entirely.

RDRAND is also a preferable source of randomness in early Unix system init, and in systems using #[no_std].

I've recently created a temporary stop gap crate hyper-thread-random to provide randomness for hyper threads with RDRAND and a fallback to ThreadRng, but it seems only logical that all things randomness-related are better 'made at home' in the rand crate.

[dhardy]

How did you manage to make your hyperlink link back to this same issue?

RDRAND support of some kind is something we want (maybe in Rand or maybe outside); @nagisa is working on a separate crate for that.

I'm less convinced about making OsRng switch to RDRAND though for a couple of reasons:

• OsRng should provide very strong random numbers; I'm not sure how much RDRAND should be trusted when it comes to cryptography
• We want a platform-agnostic design; if OsRng is super-fast on some platforms and much slower on others then something like thread_rng is still needed

We could instead let thread_rng use RDRAND when compiled with a feature flag and with platform support, or perhaps add a light_rng function or something for this.

There are some other things we could use RDRAND for but they're beyond the scope of OsRng (e.g. RDRAND could be another fallback for EntropyRng).

[raphaelcohn]

How did you manage to make your hyperlink link back to this same issue?

@dhardy I have absolutely no idea! Seems one leaves the round brackets empty.

OsRng should provide very strong random numbers; I'm not sure how much RDRAND should be trusted when it comes to cryptography

I can buy that, although I'd suggest we want to make it very clear that OsRng provides cryptographically secure random numbers. My cursory 5 minutes researching this crate before posting my issue didn't make that clear, and it is an important point. May be I skim read too quickly (;). Regardless, it's a classic problem for randomness libraries and functions - distinguishing what kind of randomness they give.

Strictly speaking, Intel suggests we use RDSEED if using random data to seed a CSPRNG, which is the main use of OsRng as I understand it. Something to borne in mind in any future decisions.

I do understand the hesitation in using an opaque random number generator from what some would perceive to be an untrustworthy source. However, there are now two sources - Intel and AMD.

We want a platform-agnostic design; if OsRng is super-fast on some platforms and much slower on others then something like thread_rng is still needed

Not sure I buy that; that's more a limitation of the platform.

We could instead let thread_rng use RDRAND when compiled with a feature flag and with platform support, or perhaps add a light_rng function or something for this.

Both of those would work the use cases I've experienced, although I think light_rnd is a better option as a downstream developer might not be able to change the feature flags at all. A processor-level random-number generator is likely to be a more common option in the near future. For instance, PowerPC ISA 3.0 (POWER9 chips only for now), which is relatively new, also now has an equivalent to RDRAND.

[dhardy]

Ultimately I'm not sure if we do want thread_rng in its current form. What we do want is:

• a generator which can be used with little overhead from any thread, is reasonably fast, statistically good, and at least not trivial to exploit; the current thread_rng does okay except for slow initialisation; RDRAND would be better
• a strong source for producing cryptographic keys; probably OsRng directly

There has been some push to make thread_rng cryptographic strength; for some uses it might make sense to have a fast thread-local very strong generator too, in which case we could potentially mix the output of the current thread_rng with RDRAND data.

So possibly just adding a light_rng function is a good approach. Perhaps @burdges has more thoughts on this?

---

I'd suggest we want to make it very clear

That's what CryptoRng is for. Hmm, not sure why OsRng doesn't implement that.

However, there are now two sources - Intel and AMD.

Plus several ARM chips, plus the PowerPC you mentioned — but we need to trust all of these to trust RDRAND, not just one of them, which is very good motivation for not using it for important keys. (Granted, this does leave plenty of other uses for RDRAND.)

that's more a limitation of the platform.

If an API behaves completely differently on different platforms then it's not a portable API. There's no good reason to relegate platforms without RDRAND to third-rate support.

[vks]

It'd be far more elegant to have perhaps a create compilation feature, or even better, when #[cfg(target_feature)] lands, to be able to have the OsRng switch over to RDRAND instead of using a syscall, and then to be able to eliminate ThreadRng set up entirely.

Why is this preferable to just using the rdrand crate?

[raphaelcohn]

If an API behaves completely differently on different platforms then it's not a portable API. There's no good reason to relegate platforms without RDRAND to third-rate support.

Oh, that's not really what I wanted to imply. Just that if some platforms have a slower random number generator than others, we shouldn't make all platforms slow. My points and yours though do suggest that there needs to be:-

• A wrapper around the 'preferred' OS or platform way of obtaining randomness;
• A cryptographically strong random number generator;
• A CPU-direct (for want of a better word for now) random number generator*
• A thread local variant (really, just a variant with minimal set-up time and minimal state) of potentially all three

All need to be (perhaps excessively so for some) clear about where they get their randomness and what they are good for and not good for.

* Then there's the argument about compile-time coding of a cpu feature vs runtime detection via cpuid. This argument can be solved by deciding what the purpose of such a random number generator is. Is it the high-performance, best-efforts but not quite cryptographically strong, or is it as a fallback for platforms missing a random number generator (eg for #[no_std] like situations)?

[raphaelcohn]

@vks That crate is X86-specific and used to use runtime detection IIRC

[nagisa]

RDRAND is not that fast. It does provide high-quality random numbers, but both throughput and latency of RDRAND is lower than pretty much anything that’s not a bare OsRng.

However, there are now two sources - Intel and AMD.

AMD is known to use a ring oscillator as a base for their RDRAND, but nobody has verified that this is exactly what’s implemented in the silicon. This is just something that AMD claims. For Intel it is still unknown what the underlying implementation is, to the best of my knowledge.

[pitdicker]

I'm not sure this issue brings anything new to the discussion, and we were already well aware of RDRAND.

The real problem seems to be @raphaelcohn that you want a fast RNG, but don't want to thread the state of the RNG through your API and don't want to use thread-local memory. As an alternative you want to use OsRng for something it isn't meant for, and turns out to be slow at.

RDRAND can be a good solution in your case, or a custom non-cryptographic RNG that lives in TLS and uses less memory.

We already have an issue about the possibility of letting ThreadRng use something else with a feature flag (or in no_std mode), #313. And that one also tracks how to make EntropyRng (which normally uses OsRng, but doesn't promise to do so) support an alternate fallback like RDAND.

But I don't think a flag to replace OsRng with RDRAND is on the table.

[raphaelcohn]

we were already well aware of RDRAND.

I was cogniscant of that before posting this issue - I even caveat-ed that in my opening paragraph "I've opened this as a new issue as I think this is something that needs proper tracking." I studied the background as best I could, and took the decision to post it. It may not have been your intention, but this comes across as snippy, and certainly puts me off wanting to engage further.

I'm not sure this issue brings anything new to the discussion

@pitdicker I mildly disagree. It's shown that the modelling of the different sources of randomness, their usefulness for different tasks, and how they can be consumed, isn't yet particularly to consumers, and that the rand crate doesn't make its prescriptions clear. In particular, it'd be nice to make clearer the purpose of ThreadRng and OsRng and EntropyRng; for example by including the quality of randomness and its source in their name, eg OperatingSystemCryptographicallySecureRandomNumberGenerator. Naming done well means a casual developer doesn't have to refer to the docs...

[dhardy]

I expect the snippiness is caused by stress trying to tie many threads together... I'm sorry if you feel put off from engaging!

I don't follow your last paragraph exactly. We are still working on the docs; on the other hand trying to avoid having to use the docs at all is not so simple. Avoiding use of acronyms altogether actually makes it harder to learn/talk about a topic in my experience, although for sure one should avoid using too many acronyms.

We will revisit ThreadRng later, so I'll probably leave the original issue until then.

[pitdicker]

It may not have been your intention, but this comes across as snippy, and certainly puts me off wanting to engage further.

Apologies. Yes, you did a good job on finding #109. My remark was not directed at you, but more that this issue is evolving into something touching many different topics that are tracked elsewhere, or are not really things that can change.

In a way we already have a tracking issue for what you want, #313. I don't know if you read it, and that of the PR, but I am afraid it won't completely explain the thoughts and problems floating around at the time.

Roughly the idea at the moment is that OsRng should simply remain an interface to the OS RNG. That is something useful to have by itself. EntropyRng is set up to be more flexible. It currently just wraps OsRng, but also has a fallback solution. The best place to add RDRAND or another solution is there, in EntropyRng.

We want to solve another problem at the same time, and that is better support for embedded. So it would be nice if RDRAND is not the only option, but to have it be user-configurable.

It's shown that the modelling of the different sources of randomness, their usefulness for different tasks, and how they can be consumed, isn't yet particularly to consumers

It is not an easy subject. We do our best to improve the documentation at the moment, so I hope things are better in the future. Different names are not going to solve everything, there are too many aspects to capture in a reasonable way in just the names.

But to come back to the motivation of this issue. I think it is better to focus this issue on finding some solution that helps with your use case, than to talk about possible redesigns of Rand (the past 9 months were already spend on that).

I believe @dhardy is playing with the idea of having some sort of light-weight variant of ThreadRng. Would that help you? Or is going the 'normal' way possible for you: to use an RNG like SmallRng and to pass references to it around?