rand: don't block before random pool is initialized #33086

Merged
merged 3 commits into from May 6, 2016

Conversation

Projects
None yet
@cardoe
Contributor

cardoe commented Apr 18, 2016

If we attempt a read with getrandom() on Linux the syscall can block
before the random pool is initialized unless the GRND_NONBLOCK flag is
passed. This flag causes getrandom() to instead return EAGAIN while the
pool is uninitialized. To avoid downstream users of crate or std
functionality that have no ability to avoid this blocking behavior this
change causes Rust to read bytes from /dev/urandom while getrandom()
would block and once getrandom() is available to use that. Fixes #32953.

Signed-off-by: Doug Goldstein cardoe@cardoe.com

@rust-highfive

This comment has been minimized.

Show comment
Hide comment
@rust-highfive

rust-highfive Apr 18, 2016

Collaborator

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

Collaborator

rust-highfive commented Apr 18, 2016

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe Apr 18, 2016

Contributor

More of a RFC than a real PR.

Contributor

cardoe commented Apr 18, 2016

More of a RFC than a real PR.

src/libstd/sys/unix/rand.rs
@@ -63,6 +65,11 @@ mod imp {
let err = errno() as libc::c_int;
if err == libc::EINTR {
continue;
+ } else if err == libc::EAGAIN {
+ let reader = File::open("/dev/urandom")?;

This comment has been minimized.

@sfackler

sfackler Apr 19, 2016

Member

getrandom_fill_bytes doesn't return a Result, so ? won't work here.

@sfackler

sfackler Apr 19, 2016

Member

getrandom_fill_bytes doesn't return a Result, so ? won't work here.

@rkruppe

This comment has been minimized.

Show comment
Hide comment
@rkruppe

rkruppe Apr 19, 2016

Contributor

Presumably this is "more of an RFC" because it could be quite controversial. If so, could you lay out the motivation and drawbacks, RFC-style (if shorter and more informally)?

I suspect the main drawback is that this may draw from a PRNG that has not yet collected sufficient entropy, which would be bad for predictability reasons. As for motivation, I've not heard complaints about getentropy blocking, since AFAIK it only happens early after booting up and never after getrandom already succeeded once. So I see little to no motivation, but surely you do?

Contributor

rkruppe commented Apr 19, 2016

Presumably this is "more of an RFC" because it could be quite controversial. If so, could you lay out the motivation and drawbacks, RFC-style (if shorter and more informally)?

I suspect the main drawback is that this may draw from a PRNG that has not yet collected sufficient entropy, which would be bad for predictability reasons. As for motivation, I've not heard complaints about getentropy blocking, since AFAIK it only happens early after booting up and never after getrandom already succeeded once. So I see little to no motivation, but surely you do?

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe Apr 19, 2016

Contributor

@rkruppe If you follow #32953 in turn docopt/docopt.rs#178 you'll see the reason is that for early boot getrandom() will block until the pool is initialized with the current Rust implementation. Rust's HashMap seeds its hasher with getrandom() since there is no knowledge if the user will be using a bounded data set or an unbounded data set. An unbounded data set would make the HashMap vulnerable to a denial of service while a bounded data set would not. The referenced tickets proposed a few solutions such as allowing crates to opt into a different hasher or a not guaranteed random seed or make the default Rust implementation a best effort. The discussion gravitated towards best effort so I tried to put together a suggestion implementing best effort.

I will also point out that the delay for an initialized random pool on some hardware and some platforms is not a non-trivial amount of time (order of minutes) so it precludes a number of Rust crates (docopt and regex are two I know about currently) from being used in Rust applications that will be used in early boot.

As far as getentropy() goes I'm not able to determine from a quick search if it blocks by default or not. But I'm willing to bet that most people are not attempting to write programs used in early boot or their init system in Rust so they are not running into this issue.

The argument against this change was that many users of HashMap do not have unbounded data sets and as a result are paying a penalty by using SipHash over FNV since FNV is more performant than SipHash (ignoring the random initializing penalty) and perhaps people should be educated about how to use HashMap for their use case and select the right hasher. The issue there is that people may not know how their crate will be used and could unintentionally expose their crate's users to a vulnerability. This argument appeared to be rejected in the favor of best effort which matches how systemd works.

Contributor

cardoe commented Apr 19, 2016

@rkruppe If you follow #32953 in turn docopt/docopt.rs#178 you'll see the reason is that for early boot getrandom() will block until the pool is initialized with the current Rust implementation. Rust's HashMap seeds its hasher with getrandom() since there is no knowledge if the user will be using a bounded data set or an unbounded data set. An unbounded data set would make the HashMap vulnerable to a denial of service while a bounded data set would not. The referenced tickets proposed a few solutions such as allowing crates to opt into a different hasher or a not guaranteed random seed or make the default Rust implementation a best effort. The discussion gravitated towards best effort so I tried to put together a suggestion implementing best effort.

I will also point out that the delay for an initialized random pool on some hardware and some platforms is not a non-trivial amount of time (order of minutes) so it precludes a number of Rust crates (docopt and regex are two I know about currently) from being used in Rust applications that will be used in early boot.

As far as getentropy() goes I'm not able to determine from a quick search if it blocks by default or not. But I'm willing to bet that most people are not attempting to write programs used in early boot or their init system in Rust so they are not running into this issue.

The argument against this change was that many users of HashMap do not have unbounded data sets and as a result are paying a penalty by using SipHash over FNV since FNV is more performant than SipHash (ignoring the random initializing penalty) and perhaps people should be educated about how to use HashMap for their use case and select the right hasher. The issue there is that people may not know how their crate will be used and could unintentionally expose their crate's users to a vulnerability. This argument appeared to be rejected in the favor of best effort which matches how systemd works.

rand: don't block before random pool is initialized
If we attempt a read with getrandom() on Linux the syscall can block
before the random pool is initialized unless the GRND_NONBLOCK flag is
passed. This flag causes getrandom() to instead return EAGAIN while the
pool is uninitialized. To avoid downstream users of crate or std
functionality that have no ability to avoid this blocking behavior this
change causes Rust to read bytes from /dev/urandom while getrandom()
would block and once getrandom() is available to use that. Fixes #32953.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Apr 19, 2016

Member

Thanks for the PR @cardoe! In terms of implementation I'd probably say that because OsRng::new returns io::Result we shouldn't have the .unwrap() for opening /dev/urandom in theory, but rather all Linux OsRng instances have an open handle to /dev/urandom (as they may need it) and then just attempt to use getrandom (nonblocking) first.

I'm gonna tag this as T-libs to ensure it comes up during triage, I'm sure there'll be some opinons!

cc @rust-lang/libs

Member

alexcrichton commented Apr 19, 2016

Thanks for the PR @cardoe! In terms of implementation I'd probably say that because OsRng::new returns io::Result we shouldn't have the .unwrap() for opening /dev/urandom in theory, but rather all Linux OsRng instances have an open handle to /dev/urandom (as they may need it) and then just attempt to use getrandom (nonblocking) first.

I'm gonna tag this as T-libs to ensure it comes up during triage, I'm sure there'll be some opinons!

cc @rust-lang/libs

@alexcrichton alexcrichton added the T-libs label Apr 19, 2016

src/libstd/sys/unix/rand.rs
+ let reader = File::open("/dev/urandom").expect("Unable to open /dev/urandom");
+ let mut reader_rng = ReaderRng::new(reader);
+ reader_rng.fill_bytes(& mut v[read..]);
+ read += v.len() as usize;

This comment has been minimized.

@bluss

bluss Apr 19, 2016

Contributor

Needs small comment with an explanation.

@bluss

bluss Apr 19, 2016

Contributor

Needs small comment with an explanation.

This comment has been minimized.

@cardoe

cardoe Apr 21, 2016

Contributor

Please let me know if 61cbd07 is sufficient and I will squash it in.

@cardoe

cardoe Apr 21, 2016

Contributor

Please let me know if 61cbd07 is sufficient and I will squash it in.

@brson

This comment has been minimized.

Show comment
Hide comment
@brson

brson Apr 19, 2016

Contributor

lgtm. I feel like the HashMap docs should say something about this, but I also don't want to make a big deal out of it. Nobody should not use HashMap because of this.

Is this also happening in the public rand crate?

Contributor

brson commented Apr 19, 2016

lgtm. I feel like the HashMap docs should say something about this, but I also don't want to make a big deal out of it. Nobody should not use HashMap because of this.

Is this also happening in the public rand crate?

@tbu-

This comment has been minimized.

Show comment
Hide comment
@tbu-

tbu- Apr 20, 2016

Contributor

"Use bad random numbers if we can't acquire good ones."

Contributor

tbu- commented Apr 20, 2016

"Use bad random numbers if we can't acquire good ones."

@tbu-

This comment has been minimized.

Show comment
Hide comment
@tbu-

tbu- Apr 20, 2016

Contributor

@alexcrichton That would block the access to random numbers on systems with getrandom if the file descriptor limit is exhausted.

Contributor

tbu- commented Apr 20, 2016

@alexcrichton That would block the access to random numbers on systems with getrandom if the file descriptor limit is exhausted.

@tbu-

This comment has been minimized.

Show comment
Hide comment
@tbu-

tbu- Apr 20, 2016

Contributor

From http://www.2uo.de/myths-about-urandom/:

/dev/urandom isn't perfect. The problems are twofold:

On Linux, unlike FreeBSD, /dev/urandom never blocks. Remember that the whole security rested on some starting randomness, a seed?

Linux's /dev/urandom happily gives you not-so-random numbers before the kernel even had the chance to gather entropy. When is that? At system start, booting the computer.

As I understand it, this pull request would mean that we sidestep the protection getrandom(2) offers and use "not-so-random" numbers for providing the program with supposedly cryptographically secure random numbers.

Contributor

tbu- commented Apr 20, 2016

From http://www.2uo.de/myths-about-urandom/:

/dev/urandom isn't perfect. The problems are twofold:

On Linux, unlike FreeBSD, /dev/urandom never blocks. Remember that the whole security rested on some starting randomness, a seed?

Linux's /dev/urandom happily gives you not-so-random numbers before the kernel even had the chance to gather entropy. When is that? At system start, booting the computer.

As I understand it, this pull request would mean that we sidestep the protection getrandom(2) offers and use "not-so-random" numbers for providing the program with supposedly cryptographically secure random numbers.

cardoe added some commits Apr 20, 2016

HashMap: add info to docs about random seed quality
The random functions that HashMap use make no guarantees about the
quality of random data so this documents that to the user so that they
are aware. This was brought about by the change to the Linux random code
to not block until the urandom pool was initialized to avoid users of
crates that internally use HashMap being caught unaware and having their
application block until the urandom pool is initialized.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
rand: add comments about getrandom() fallback
Add some comments so that people know why we are performing a fallback
from getrandom() and what that fallback aims to achieve.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe Apr 21, 2016

Contributor

@brson I've added 121225f which hopefully adds some good user visible information in HashMap. Let me know if that's good and I can squash it in.

As far as the public rand crate goes, I was waiting to see how this progressed but planned on submitting something to that as well.

Contributor

cardoe commented Apr 21, 2016

@brson I've added 121225f which hopefully adds some good user visible information in HashMap. Let me know if that's good and I can squash it in.

As far as the public rand crate goes, I was waiting to see how this progressed but planned on submitting something to that as well.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe Apr 21, 2016

Contributor

@tbu- I'd recommend reading my long explanation that I gave @rkruppe above and followed the two referenced issues. I've rehashed the reasons and the options forward in ways I feel are pretty clear between both tickets and this PR. This PR implements the option that seemed favorable by the Rust maintainers. Obviously if a different option is deemed more favorable I will implement that as well.

Contributor

cardoe commented Apr 21, 2016

@tbu- I'd recommend reading my long explanation that I gave @rkruppe above and followed the two referenced issues. I've rehashed the reasons and the options forward in ways I feel are pretty clear between both tickets and this PR. This PR implements the option that seemed favorable by the Rust maintainers. Obviously if a different option is deemed more favorable I will implement that as well.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe Apr 21, 2016

Contributor

@alexcrichton I see two issues with always having /dev/urandom opened is we're consuming a fd from a process always which might not be desirable. And we're back to being dependent on having /dev/urandom be open-able which one of the reasons people tried to get away from using it, specifically the LibreSSL people. See https://lwn.net/Articles/606141/ for some context.

Contributor

cardoe commented Apr 21, 2016

@alexcrichton I see two issues with always having /dev/urandom opened is we're consuming a fd from a process always which might not be desirable. And we're back to being dependent on having /dev/urandom be open-able which one of the reasons people tried to get away from using it, specifically the LibreSSL people. See https://lwn.net/Articles/606141/ for some context.

@bluss

This comment has been minimized.

Show comment
Hide comment
@bluss

bluss Apr 21, 2016

Contributor

@tbu- The only use for this Rng is keys for the HashMap hash function.

Contributor

bluss commented Apr 21, 2016

@tbu- The only use for this Rng is keys for the HashMap hash function.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 2, 2016

Contributor

@alexcrichton Does that reasoning make sense as to why we don't want to always have /dev/urandom opened? @brson Do the two addon commits address your review adequately?

Contributor

cardoe commented May 2, 2016

@alexcrichton Does that reasoning make sense as to why we don't want to always have /dev/urandom opened? @brson Do the two addon commits address your review adequately?

@sfackler

This comment has been minimized.

Show comment
Hide comment
@sfackler

sfackler May 2, 2016

Member

We might want to consider expanding our set of platform specific versions of this logic - we could use SecRandomCopyBytes on iOS and OSX for example.

Member

sfackler commented May 2, 2016

We might want to consider expanding our set of platform specific versions of this logic - we could use SecRandomCopyBytes on iOS and OSX for example.

@tbu-

This comment has been minimized.

Show comment
Hide comment
@tbu-

tbu- May 2, 2016

Contributor

@bluss Fair enough. But it should probably still be pointed out that we use low-quality random numbers for the hash map if there aren't any high-quality random numbers available.

I don't know how that affects the attack surface, but generally Rust had chosen the most secure variant that is still practical (for hashmaps), e.g. choosing a secure hasher over a fast one.

Contributor

tbu- commented May 2, 2016

@bluss Fair enough. But it should probably still be pointed out that we use low-quality random numbers for the hash map if there aren't any high-quality random numbers available.

I don't know how that affects the attack surface, but generally Rust had chosen the most secure variant that is still practical (for hashmaps), e.g. choosing a secure hasher over a fast one.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 2, 2016

Contributor

@sfackler The code already does that. Look farther down in the source file.

Contributor

cardoe commented May 2, 2016

@sfackler The code already does that. Look farther down in the source file.

@sfackler

This comment has been minimized.

Show comment
Hide comment
@sfackler

sfackler May 2, 2016

Member

Oh great!

Member

sfackler commented May 2, 2016

Oh great!

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 2, 2016

Contributor

@tbu- I added documentation to that effect in 121225f, are you not happy with the wording? The attack surface is not really affected because the only time a hash map is vulnerable is if the data set is unbounded. Most of the time applications that allow unbounded data sets (e.g. network daemons) are going to be started up after enough hardware (an interrupt source such as a network card). The result here is that it doesn't matter. Plus a weaker random value would still have to be guessed by someone over the network to be able to attack the hash map. Honestly I would recommend doing some reading about the attack surface of hash maps, what and how they need to be seeded to be secure in those cases, the Linux random number generator implementation and the use cases of how this code will be used because you're honestly spreading a little bit of FUD here.

Contributor

cardoe commented May 2, 2016

@tbu- I added documentation to that effect in 121225f, are you not happy with the wording? The attack surface is not really affected because the only time a hash map is vulnerable is if the data set is unbounded. Most of the time applications that allow unbounded data sets (e.g. network daemons) are going to be started up after enough hardware (an interrupt source such as a network card). The result here is that it doesn't matter. Plus a weaker random value would still have to be guessed by someone over the network to be able to attack the hash map. Honestly I would recommend doing some reading about the attack surface of hash maps, what and how they need to be seeded to be secure in those cases, the Linux random number generator implementation and the use cases of how this code will be used because you're honestly spreading a little bit of FUD here.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton May 2, 2016

Member

@cardoe

Does that reasoning make sense as to why we don't want to always have /dev/urandom opened?

Yes, albeit a seemingly much bigger hammer than is necessary here for just a pair of keys to hash maps.

In any case, though, the PR looks ok to me, just wanna make sure that others in @rust-lang/libs are also signed off.

Member

alexcrichton commented May 2, 2016

@cardoe

Does that reasoning make sense as to why we don't want to always have /dev/urandom opened?

Yes, albeit a seemingly much bigger hammer than is necessary here for just a pair of keys to hash maps.

In any case, though, the PR looks ok to me, just wanna make sure that others in @rust-lang/libs are also signed off.

@brson

This comment has been minimized.

Show comment
Hide comment
@brson

brson May 3, 2016

Contributor

@cardoe Yes, thank you.

Contributor

brson commented May 3, 2016

@cardoe Yes, thank you.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton May 5, 2016

Member

The libs team discussed this PR during triage today and the decision was to merge. We may want to comment the internal rand library to indicate that it's not suitable for anything but what the standard library needs (for implementation details such as this). We also discussed panicking here if opening /dev/urandom fails, and the conclusion was that this is probably fine for now and we can revisit later if need be.

Thanks again for the PR @cardoe!

@bors: r+ 61cbd07

Member

alexcrichton commented May 5, 2016

The libs team discussed this PR during triage today and the decision was to merge. We may want to comment the internal rand library to indicate that it's not suitable for anything but what the standard library needs (for implementation details such as this). We also discussed panicking here if opening /dev/urandom fails, and the conclusion was that this is probably fine for now and we can revisit later if need be.

Thanks again for the PR @cardoe!

@bors: r+ 61cbd07

@tbu-

This comment has been minimized.

Show comment
Hide comment
@tbu-

tbu- May 5, 2016

Contributor

@alexcrichton Are you aware of the fact that OsRng is directly exposed to users of the standard library via std::rand::OsRng and used to provide cryptographically random numbers before this commit? Maybe only the random number generation for HashMaps should be changed.

EDIT: This is incorrect, see @huonw below.

Contributor

tbu- commented May 5, 2016

@alexcrichton Are you aware of the fact that OsRng is directly exposed to users of the standard library via std::rand::OsRng and used to provide cryptographically random numbers before this commit? Maybe only the random number generation for HashMaps should be changed.

EDIT: This is incorrect, see @huonw below.

@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors May 5, 2016

Contributor

⌛️ Testing commit 61cbd07 with merge eafcbd1...

Contributor

bors commented May 5, 2016

⌛️ Testing commit 61cbd07 with merge eafcbd1...

bors added a commit that referenced this pull request May 5, 2016

Auto merge of #33086 - cardoe:non-blocking-rand-read, r=alexcrichton
rand: don't block before random pool is initialized

If we attempt a read with getrandom() on Linux the syscall can block
before the random pool is initialized unless the GRND_NONBLOCK flag is
passed. This flag causes getrandom() to instead return EAGAIN while the
pool is uninitialized. To avoid downstream users of crate or std
functionality that have no ability to avoid this blocking behavior this
change causes Rust to read bytes from /dev/urandom while getrandom()
would block and once getrandom() is available to use that. Fixes #32953.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
@huonw

This comment has been minimized.

Show comment
Hide comment
@huonw

huonw May 5, 2016

Member

@tbu- AFAICT, it is private, and would be unstable if it were public; could you clarify what you mean?

Member

huonw commented May 5, 2016

@tbu- AFAICT, it is private, and would be unstable if it were public; could you clarify what you mean?

@tbu-

This comment has been minimized.

Show comment
Hide comment
@tbu-

tbu- May 5, 2016

Contributor

@huonw Ah right. Sorry, I messed up here, I didn't see that mod rand isn't exported by std. I thought we were exposing random number generators previously and now changed them to yield not-so-random numbers.

Contributor

tbu- commented May 5, 2016

@huonw Ah right. Sorry, I messed up here, I didn't see that mod rand isn't exported by std. I thought we were exposing random number generators previously and now changed them to yield not-so-random numbers.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 5, 2016

Contributor

@tbu- The point of this change was to keep the scope limited just to the internal part of std (e.g. HashMap) and not exported for use. If I changed the rand crate then yes it would be exposed to other users. But again the scope of the change is extremely limited, even if we did change the rand crate. I really encourage you to read about how the Linux getrandom() syscall works, the Linux random number generator and follow the code flow of the fallbacks and see that once the urandom pool is initialized this change has no effect.

Contributor

cardoe commented May 5, 2016

@tbu- The point of this change was to keep the scope limited just to the internal part of std (e.g. HashMap) and not exported for use. If I changed the rand crate then yes it would be exposed to other users. But again the scope of the change is extremely limited, even if we did change the rand crate. I really encourage you to read about how the Linux getrandom() syscall works, the Linux random number generator and follow the code flow of the fallbacks and see that once the urandom pool is initialized this change has no effect.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 5, 2016

Contributor

@alexcrichton Thank you! If anyone on the libs team wants to see a behavior change here I will gladly do an update.

Contributor

cardoe commented May 5, 2016

@alexcrichton Thank you! If anyone on the libs team wants to see a behavior change here I will gladly do an update.

@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors May 5, 2016

Contributor

💔 Test failed - auto-win-gnu-64-opt

Contributor

bors commented May 5, 2016

💔 Test failed - auto-win-gnu-64-opt

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton May 5, 2016

Member

@bors: retry

On Thu, May 5, 2016 at 7:59 AM, bors notifications@github.com wrote:

[image: 💔] Test failed - auto-win-gnu-64-opt
http://buildbot.rust-lang.org/builders/auto-win-gnu-64-opt/builds/4086


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#33086 (comment)

Member

alexcrichton commented May 5, 2016

@bors: retry

On Thu, May 5, 2016 at 7:59 AM, bors notifications@github.com wrote:

[image: 💔] Test failed - auto-win-gnu-64-opt
http://buildbot.rust-lang.org/builders/auto-win-gnu-64-opt/builds/4086


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#33086 (comment)

@bors

This comment has been minimized.

Show comment
Hide comment
@bors

bors May 6, 2016

Contributor

⌛️ Testing commit 61cbd07 with merge a36c419...

Contributor

bors commented May 6, 2016

⌛️ Testing commit 61cbd07 with merge a36c419...

bors added a commit that referenced this pull request May 6, 2016

Auto merge of #33086 - cardoe:non-blocking-rand-read, r=alexcrichton
rand: don't block before random pool is initialized

If we attempt a read with getrandom() on Linux the syscall can block
before the random pool is initialized unless the GRND_NONBLOCK flag is
passed. This flag causes getrandom() to instead return EAGAIN while the
pool is uninitialized. To avoid downstream users of crate or std
functionality that have no ability to avoid this blocking behavior this
change causes Rust to read bytes from /dev/urandom while getrandom()
would block and once getrandom() is available to use that. Fixes #32953.

Signed-off-by: Doug Goldstein <cardoe@cardoe.com>
@bors

@bors bors merged commit 61cbd07 into rust-lang:master May 6, 2016

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details
@eternaleye

This comment has been minimized.

Show comment
Hide comment
@eternaleye

eternaleye May 9, 2016

Contributor

I realize I'm a bit late on this, but I wanted to explain why I feel this is a poor decision, and one based on a false dichotomy.

The problem as expressed is (to my understanding):

  • HashMap initialization requires randomness for initializing SipHash, to avoid hash-DOS
  • Acquiring randomness may block during early boot when using getrandom()
  • Thus, using HashMap during early boot may cause blocking
  • Thus, Rust is ill-prepared for early-boot environments

This is justified by that in the fallback case to /dev/urandom, such blocking does not occur, and thus the security benefits of it can't be guaranteed.

However:

  • Security is best served not by taking the common subset of behaviors, but by choosing the most secure option available at any given time. If this was not true, getrandom() would never have blocked in the first place. As "getrandom() does not exist" returns a distinct error from "the entropy pool is not initialized", I feel the justification is specious.
  • HashMap is not the only map available, and BTreeMap does not require randomness to avoid DOS.
  • Early-boot environments are special; expecting arbitrary code written for normal environments to be sensible for early-boot unmodified is fallacious. Thus, recommending the use of a different map is reasonable.
  • The weak initialization has ramifications well beyond early boot; consider services which must start early and continue running - init systems, DBus, FUSE filesystems, etc. These then become more vulnerable to hash-DOS over the entire runtime of the system, and are crucial system components.
  • There's no way for the hash initialization, if initially weak, to become strong via reseeding later.

In essence, I feel that this change weakens HashMap for the explicit purpose of supporting an antipattern with detrimental effects on the ecosystem, when the proper way of handling the issue would be to recommend using BTreeMap if entropy may not be available.

EDIT: In addition, on the BSDs, HashMap initialization will still block, so this change only even accomplishes its stated (and as I explain above, IMO mistaken) goals on Linux. For early-boot usage on BSD systems, my proposed approach of using BTreeMap instead would still be needed.

Contributor

eternaleye commented May 9, 2016

I realize I'm a bit late on this, but I wanted to explain why I feel this is a poor decision, and one based on a false dichotomy.

The problem as expressed is (to my understanding):

  • HashMap initialization requires randomness for initializing SipHash, to avoid hash-DOS
  • Acquiring randomness may block during early boot when using getrandom()
  • Thus, using HashMap during early boot may cause blocking
  • Thus, Rust is ill-prepared for early-boot environments

This is justified by that in the fallback case to /dev/urandom, such blocking does not occur, and thus the security benefits of it can't be guaranteed.

However:

  • Security is best served not by taking the common subset of behaviors, but by choosing the most secure option available at any given time. If this was not true, getrandom() would never have blocked in the first place. As "getrandom() does not exist" returns a distinct error from "the entropy pool is not initialized", I feel the justification is specious.
  • HashMap is not the only map available, and BTreeMap does not require randomness to avoid DOS.
  • Early-boot environments are special; expecting arbitrary code written for normal environments to be sensible for early-boot unmodified is fallacious. Thus, recommending the use of a different map is reasonable.
  • The weak initialization has ramifications well beyond early boot; consider services which must start early and continue running - init systems, DBus, FUSE filesystems, etc. These then become more vulnerable to hash-DOS over the entire runtime of the system, and are crucial system components.
  • There's no way for the hash initialization, if initially weak, to become strong via reseeding later.

In essence, I feel that this change weakens HashMap for the explicit purpose of supporting an antipattern with detrimental effects on the ecosystem, when the proper way of handling the issue would be to recommend using BTreeMap if entropy may not be available.

EDIT: In addition, on the BSDs, HashMap initialization will still block, so this change only even accomplishes its stated (and as I explain above, IMO mistaken) goals on Linux. For early-boot usage on BSD systems, my proposed approach of using BTreeMap instead would still be needed.

@nejucomo

This comment has been minimized.

Show comment
Hide comment
@nejucomo

nejucomo May 10, 2016

Chiming in to heartily concur with @eternaleye 's post. Also imagine you are a new-to-rust developer and you read this patch's HashMap documentation:

This means that the ordering of the keys is randomized, but makes the tables more resistant to denial-of-service attacks (Hash DoS). No guarantees are made to the quality of the random data. The implementation uses the best available random data from your platform at the time of creation. This behavior can be overridden with one of the constructors.

Now, I either need to be concerned about DoS for my use case, or I do not. If I do care, how can I evaluate whether or not HashMap helps my use case?

If the documentation were extended to spell out the conditions in which the entropy (and thus DoS) protection is weak, can I determine whether or not my use case is in that condition? (A panic is one clear way to determine if that's the case, and for the use cases that require HashMap DoS protection, this provides fail-closed security.)

If it's not possible to evaluate or determine when or how this DoS protection comes into effect, then isn't its value largely diminished? That's akin to ignoring if your HTTPS connection has a valid cert or not because "encryption may help prevent network eavesdroppers".

Meanwhile, for the other use case of early startup, if DoS protection does not matter, then why use a library that sometimes provides that feature anyway, instead of a simpler alternative?

Chiming in to heartily concur with @eternaleye 's post. Also imagine you are a new-to-rust developer and you read this patch's HashMap documentation:

This means that the ordering of the keys is randomized, but makes the tables more resistant to denial-of-service attacks (Hash DoS). No guarantees are made to the quality of the random data. The implementation uses the best available random data from your platform at the time of creation. This behavior can be overridden with one of the constructors.

Now, I either need to be concerned about DoS for my use case, or I do not. If I do care, how can I evaluate whether or not HashMap helps my use case?

If the documentation were extended to spell out the conditions in which the entropy (and thus DoS) protection is weak, can I determine whether or not my use case is in that condition? (A panic is one clear way to determine if that's the case, and for the use cases that require HashMap DoS protection, this provides fail-closed security.)

If it's not possible to evaluate or determine when or how this DoS protection comes into effect, then isn't its value largely diminished? That's akin to ignoring if your HTTPS connection has a valid cert or not because "encryption may help prevent network eavesdroppers".

Meanwhile, for the other use case of early startup, if DoS protection does not matter, then why use a library that sometimes provides that feature anyway, instead of a simpler alternative?

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton May 10, 2016

Member

@eternaleye this was why we concluded that librand should be documented as not being suitable outside the standard library.

Member

alexcrichton commented May 10, 2016

@eternaleye this was why we concluded that librand should be documented as not being suitable outside the standard library.

@eternaleye

This comment has been minimized.

Show comment
Hide comment
@eternaleye

eternaleye May 10, 2016

Contributor

@alexcrichton: But that doesn't address my point: This change directly encourages using HashMap in early-boot environments, which (for code which continues running after early-boot) means it retains a vulnerability to hash-DOS. Using librand outside std has nothing to do with it; using HashMap with insufficient entropy is the problem.

Contributor

eternaleye commented May 10, 2016

@alexcrichton: But that doesn't address my point: This change directly encourages using HashMap in early-boot environments, which (for code which continues running after early-boot) means it retains a vulnerability to hash-DOS. Using librand outside std has nothing to do with it; using HashMap with insufficient entropy is the problem.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 10, 2016

Contributor

@eternaleye Many things rely on HashMap internally. How do you propose that people handle this situation to avoid that hidden dependency on HashMap? Additionally how do you propose that users are inherently aware of this and can avoid it? You mention init systems in your post so I'll link you to systemd, systemd/systemd@97768fc which has had the same behavior since 2014. All the other daemons you mentioned would come up after the random pool is initialized in most use cases and render your arguments invalid.

@nejucomo Hash tables that accept unbound data sets (where a user can put any data in) are vulnerable to DoS attacks if they are not seeded.

While the original bug report asked for some crates to move away from using HashMap and then later on asked that an application can decide how HashMap's are seeded this was the solution that was accepted.

@eternaleye

But that doesn't address my point: This change directly encourages using HashMap in early-boot environments, which (for code which continues running after early-boot) means it retains a vulnerability to hash-DOS. Using librand outside std has nothing to do with it; using HashMap with insufficient entropy is the problem.

You assume that every daemon is accepted unbounded data that is started before the random pool is initialized. You also assume that existing crate authors are aware of this and have the ability to avoid using HashMap. There is almost no ability with the current Rust crate environment to write a program past "Hello World" that can be used without a HashMap wanting to be initialized. I would hardly call this "directly encouraging using HashMap".

Honestly I would love to see the people opposed to this change propose a solution that's acceptable to the Rust team because I would be willing to implement it. Just like I was willing to implement the solution that was requested of me initially.

Contributor

cardoe commented May 10, 2016

@eternaleye Many things rely on HashMap internally. How do you propose that people handle this situation to avoid that hidden dependency on HashMap? Additionally how do you propose that users are inherently aware of this and can avoid it? You mention init systems in your post so I'll link you to systemd, systemd/systemd@97768fc which has had the same behavior since 2014. All the other daemons you mentioned would come up after the random pool is initialized in most use cases and render your arguments invalid.

@nejucomo Hash tables that accept unbound data sets (where a user can put any data in) are vulnerable to DoS attacks if they are not seeded.

While the original bug report asked for some crates to move away from using HashMap and then later on asked that an application can decide how HashMap's are seeded this was the solution that was accepted.

@eternaleye

But that doesn't address my point: This change directly encourages using HashMap in early-boot environments, which (for code which continues running after early-boot) means it retains a vulnerability to hash-DOS. Using librand outside std has nothing to do with it; using HashMap with insufficient entropy is the problem.

You assume that every daemon is accepted unbounded data that is started before the random pool is initialized. You also assume that existing crate authors are aware of this and have the ability to avoid using HashMap. There is almost no ability with the current Rust crate environment to write a program past "Hello World" that can be used without a HashMap wanting to be initialized. I would hardly call this "directly encouraging using HashMap".

Honestly I would love to see the people opposed to this change propose a solution that's acceptable to the Rust team because I would be willing to implement it. Just like I was willing to implement the solution that was requested of me initially.

@eternaleye

This comment has been minimized.

Show comment
Hide comment
@eternaleye

eternaleye May 10, 2016

Contributor

@cardoe: Many things use /proc, or /sys, or other things that may not be available at early-boot internally. Call fexecve(3) on a kernel without execveat(2) (anything prior to 3.19) without /proc mounted? Boom. Kernel has execveat(2) but glibc is too old to use it? Boom.

Early boot is a constrained environment that requires significant care; papering over "HashMap may block" by changing it to "HashMap may be vulnerable to DOS" doesn't fix the issue. Instead, it allows the issue's ramifications to propagate beyond early boot.

And yes, systemd does something I consider incorrect there, especially as it does allow unbounded data: systemctl set-environment among other things. (like the DBus interface for defining transient units...)

Contributor

eternaleye commented May 10, 2016

@cardoe: Many things use /proc, or /sys, or other things that may not be available at early-boot internally. Call fexecve(3) on a kernel without execveat(2) (anything prior to 3.19) without /proc mounted? Boom. Kernel has execveat(2) but glibc is too old to use it? Boom.

Early boot is a constrained environment that requires significant care; papering over "HashMap may block" by changing it to "HashMap may be vulnerable to DOS" doesn't fix the issue. Instead, it allows the issue's ramifications to propagate beyond early boot.

And yes, systemd does something I consider incorrect there, especially as it does allow unbounded data: systemctl set-environment among other things. (like the DBus interface for defining transient units...)

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 10, 2016

Contributor

@eternaleye Enough grandstanding and throwing out irrelevant examples. Propose an actual solution to the issue at hand and we can move forward. I am more than willing to implement another solution.

Contributor

cardoe commented May 10, 2016

@eternaleye Enough grandstanding and throwing out irrelevant examples. Propose an actual solution to the issue at hand and we can move forward. I am more than willing to implement another solution.

@eternaleye

This comment has been minimized.

Show comment
Hide comment
@eternaleye

eternaleye May 10, 2016

Contributor

@cardoe: I am not grandstanding, and I'm rather insulted by the implication. In addition, the examples are not irrelevant. Fundamentally, early boot is not a POSIX environment, and it makes no guarantees whatsoever of what is available. std relies on more than early boot provides.

/dev/urandom may not exist, as the device node may not have been created. Various C library calls may not work, because /proc and /sys are not mounted. The list goes on.

In addition, while blocking when unable to get randomness makes HashMap itself unusable in early boot, accepting sub-par randomness allows using HashMap in early boot to silently reduce the security of the system later in its runtime, after it's fully initialized.

Fundamentally, early boot is a #![no_std] environment - some crates like alloc are usable, but others are not. This "solution" isn't - it's an ad-hoc patch job for a very small part of a larger issue, with real and problematic side effects.

Furthermore, this ignores that not all the world is Linux. Different systems have different behaviors in an early-boot environment. For instance, early boot BSDs block on /dev/urandom, and thus have the behavior of getrandom() in both cases. The user space random functions on BSDs seed themselves from the kernel random, and thus block as well.

Contributor

eternaleye commented May 10, 2016

@cardoe: I am not grandstanding, and I'm rather insulted by the implication. In addition, the examples are not irrelevant. Fundamentally, early boot is not a POSIX environment, and it makes no guarantees whatsoever of what is available. std relies on more than early boot provides.

/dev/urandom may not exist, as the device node may not have been created. Various C library calls may not work, because /proc and /sys are not mounted. The list goes on.

In addition, while blocking when unable to get randomness makes HashMap itself unusable in early boot, accepting sub-par randomness allows using HashMap in early boot to silently reduce the security of the system later in its runtime, after it's fully initialized.

Fundamentally, early boot is a #![no_std] environment - some crates like alloc are usable, but others are not. This "solution" isn't - it's an ad-hoc patch job for a very small part of a larger issue, with real and problematic side effects.

Furthermore, this ignores that not all the world is Linux. Different systems have different behaviors in an early-boot environment. For instance, early boot BSDs block on /dev/urandom, and thus have the behavior of getrandom() in both cases. The user space random functions on BSDs seed themselves from the kernel random, and thus block as well.

@briansmith

This comment has been minimized.

Show comment
Hide comment
@briansmith

briansmith May 10, 2016

Propose an actual solution to the issue at hand and we can move forward. I am more than willing to implement another solution.

  • By default OsRng should just use getrandom and not fall back to reading from /dev/urandom unless getrandom returns ENOSYS. i.e. by default, the old blocking behavior is restored.
  • Much like the way the application can configure panic! to abort on failure, a similar mechanism should be implemented where the application (and in particular not a library) can say that OsRng should never block. This mechanism would be intended only for use by specialized apps that are designed to run before the OS ensures that getrandom() would never block.

Note that such design follows the "secure by default" principle.

In particular, consider an application that invokes the getrandom syscall directly and, if the system doesn't return ENOSYS, filters out the open[at], read, etc. syscalls using seccomp-bpf. Such an application would work correctly with the previous behavior but not with the currently-commited behavior. (Thus, the already-committed change in this PR is a backward compatibility regression.) With the strategy I suggested above, the backward compatibility would be restored.

Propose an actual solution to the issue at hand and we can move forward. I am more than willing to implement another solution.

  • By default OsRng should just use getrandom and not fall back to reading from /dev/urandom unless getrandom returns ENOSYS. i.e. by default, the old blocking behavior is restored.
  • Much like the way the application can configure panic! to abort on failure, a similar mechanism should be implemented where the application (and in particular not a library) can say that OsRng should never block. This mechanism would be intended only for use by specialized apps that are designed to run before the OS ensures that getrandom() would never block.

Note that such design follows the "secure by default" principle.

In particular, consider an application that invokes the getrandom syscall directly and, if the system doesn't return ENOSYS, filters out the open[at], read, etc. syscalls using seccomp-bpf. Such an application would work correctly with the previous behavior but not with the currently-commited behavior. (Thus, the already-committed change in this PR is a backward compatibility regression.) With the strategy I suggested above, the backward compatibility would be restored.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 10, 2016

Contributor

@eternaleye

Fundamentally, early boot is not a POSIX environment, and it makes no guarantees whatsoever of what is available. std relies on more than early boot provides.

Where in the world do you get this crazy notion from? I'd love to see some document to back this assertion up.

/dev/urandom may not exist, as the device node may not have been created. Various C library calls may not work, because /proc and /sys are not mounted. The list goes on.

So you're arguing that Rust can not be used as an init system then? So Rust is acceptable to bare metal embedded systems and to fully stood up POSIX environments only.

Fundamentally, early boot is a #![no_std] environment - some crates like alloc are usable, but others are not. This "solution" isn't - it's an ad-hoc patch job for a very small part of a larger issue, with real and problematic side effects.

That's absolutely absurd assertion to make. #![no_std] is clearly not for that use case to imply it is absolutely false. I encourage you to read the Rust book again. https://doc.rust-lang.org/book/no-stdlib.html

The Rust developers pointed me in this direction because users have the ability to use a different hash function with HashMap should they need/want to. But users of basic crates that provide things like argument parsing for example have no way to specify the hash function used for the HashMap implementation. The point here is you can opt-in to what you need.

@briansmith
Saying "revert the change" does not count as proposing a solution. Plain and simple if you need to spew unbound data at HashMap and you are concerned that your application is started before the random pool is initialized use the existing mechanisms to supply your own hasher function. I will gladly update the documentation to reflect that.

Contributor

cardoe commented May 10, 2016

@eternaleye

Fundamentally, early boot is not a POSIX environment, and it makes no guarantees whatsoever of what is available. std relies on more than early boot provides.

Where in the world do you get this crazy notion from? I'd love to see some document to back this assertion up.

/dev/urandom may not exist, as the device node may not have been created. Various C library calls may not work, because /proc and /sys are not mounted. The list goes on.

So you're arguing that Rust can not be used as an init system then? So Rust is acceptable to bare metal embedded systems and to fully stood up POSIX environments only.

Fundamentally, early boot is a #![no_std] environment - some crates like alloc are usable, but others are not. This "solution" isn't - it's an ad-hoc patch job for a very small part of a larger issue, with real and problematic side effects.

That's absolutely absurd assertion to make. #![no_std] is clearly not for that use case to imply it is absolutely false. I encourage you to read the Rust book again. https://doc.rust-lang.org/book/no-stdlib.html

The Rust developers pointed me in this direction because users have the ability to use a different hash function with HashMap should they need/want to. But users of basic crates that provide things like argument parsing for example have no way to specify the hash function used for the HashMap implementation. The point here is you can opt-in to what you need.

@briansmith
Saying "revert the change" does not count as proposing a solution. Plain and simple if you need to spew unbound data at HashMap and you are concerned that your application is started before the random pool is initialized use the existing mechanisms to supply your own hasher function. I will gladly update the documentation to reflect that.

@briansmith

This comment has been minimized.

Show comment
Hide comment
@briansmith

briansmith May 10, 2016

@briansmith
Saying "revert the change" does not count as proposing a solution. Plain and simple if you need to spew unbound data at HashMap and you are concerned that your application is started before the random pool is initialized use the existing mechanisms to supply your own hasher function. I will gladly update the documentation to reflect that.

The first bullet of what I suggested was indeed effectively "revert the change." Such reversion is needed to undo the compatibility regression anyway. However, the second bullet of what I suggested was the necessary step to get the behavior you want: an opt-in mechanism that any application can make to get the behavior you want.

@briansmith
Saying "revert the change" does not count as proposing a solution. Plain and simple if you need to spew unbound data at HashMap and you are concerned that your application is started before the random pool is initialized use the existing mechanisms to supply your own hasher function. I will gladly update the documentation to reflect that.

The first bullet of what I suggested was indeed effectively "revert the change." Such reversion is needed to undo the compatibility regression anyway. However, the second bullet of what I suggested was the necessary step to get the behavior you want: an opt-in mechanism that any application can make to get the behavior you want.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 10, 2016

Contributor

As I've said on the previous issues. Provide another hasher function in the standard library and steer crate authors to the right implementation for the right use case. I get the "secure by default" attitude but people would similarly find it over the top to establish a full TLS connection and teardown for 2 bytes of data. Its all about the right tool for the right job. In most of the crates I see using HashMap, using a fully seeded SipHash is over the top.

Contributor

cardoe commented May 10, 2016

As I've said on the previous issues. Provide another hasher function in the standard library and steer crate authors to the right implementation for the right use case. I get the "secure by default" attitude but people would similarly find it over the top to establish a full TLS connection and teardown for 2 bytes of data. Its all about the right tool for the right job. In most of the crates I see using HashMap, using a fully seeded SipHash is over the top.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 10, 2016

Contributor

@briansmith

The first bullet of what I suggested was indeed effectively "revert the change." Such reversion is needed to undo the compatibility regression anyway. However, the second bullet of what I suggested was the necessary step to get the behavior you want: an opt-in mechanism that any application can make to get the behavior you want.

As was pointed out by @alexcrichton that was discussed and the team decided to proceed forward.

Contributor

cardoe commented May 10, 2016

@briansmith

The first bullet of what I suggested was indeed effectively "revert the change." Such reversion is needed to undo the compatibility regression anyway. However, the second bullet of what I suggested was the necessary step to get the behavior you want: an opt-in mechanism that any application can make to get the behavior you want.

As was pointed out by @alexcrichton that was discussed and the team decided to proceed forward.

@eternaleye

This comment has been minimized.

Show comment
Hide comment
@eternaleye

eternaleye May 10, 2016

Contributor

@cardoe: You are putting words in my mouth. Please stop that.

Where in the world do you get this crazy notion from?

...the fact that various things required by POSIX can be absent during early boot?

  • Mandatory utilities such as basename(1) may not be present (consider initramfs)
  • /dev may not contain /dev/console or /dev/tty, which POSIX mandates

So you're arguing that Rust can not be used as an init system then? So Rust is acceptable to bare metal embedded systems and to fully stood up POSIX environments only.

No, I said what I meant and I meant what I said. Some crates are valid and some are not. std assumes all of its component crates are valid; #![no_std] allows more fine-grained handling. The very link you gave bears this up, starting with the very first paragraph:

Rust’s standard library [...] assumes support for various features of its host system [...] There are systems that do not have these features, however, and Rust can work with those too! To do so, we tell Rust that we don’t want to use the standard library via an attribute: #![no_std].

One can then load the individual crates that are usually bundled behind the libstd facade: liballoc (which provides Box<T>, Rc<T>, and more), libcontainers (which provides, among other things, BTreeMap, HashMap, etc), and so forth.

#![no_std] is in no way only "acceptable to bare metal embedded systems" - that's flat out incorrect.

It's not black and white "std or nothing" - it's "open up the std black box and only use what's needed."

Unless the full set of assumptions behind std are upheld, the facade is not the correct abstraction to use - but the individual crates behind the facade have simpler assumptions, and can be used even if the full power of std is off the table.

Contributor

eternaleye commented May 10, 2016

@cardoe: You are putting words in my mouth. Please stop that.

Where in the world do you get this crazy notion from?

...the fact that various things required by POSIX can be absent during early boot?

  • Mandatory utilities such as basename(1) may not be present (consider initramfs)
  • /dev may not contain /dev/console or /dev/tty, which POSIX mandates

So you're arguing that Rust can not be used as an init system then? So Rust is acceptable to bare metal embedded systems and to fully stood up POSIX environments only.

No, I said what I meant and I meant what I said. Some crates are valid and some are not. std assumes all of its component crates are valid; #![no_std] allows more fine-grained handling. The very link you gave bears this up, starting with the very first paragraph:

Rust’s standard library [...] assumes support for various features of its host system [...] There are systems that do not have these features, however, and Rust can work with those too! To do so, we tell Rust that we don’t want to use the standard library via an attribute: #![no_std].

One can then load the individual crates that are usually bundled behind the libstd facade: liballoc (which provides Box<T>, Rc<T>, and more), libcontainers (which provides, among other things, BTreeMap, HashMap, etc), and so forth.

#![no_std] is in no way only "acceptable to bare metal embedded systems" - that's flat out incorrect.

It's not black and white "std or nothing" - it's "open up the std black box and only use what's needed."

Unless the full set of assumptions behind std are upheld, the facade is not the correct abstraction to use - but the individual crates behind the facade have simpler assumptions, and can be used even if the full power of std is off the table.

@cardoe

This comment has been minimized.

Show comment
Hide comment
@cardoe

cardoe May 10, 2016

Contributor

@eternaleye

...the fact that various things required by POSIX can be absent during early boot?

Mandatory utilities such as basename(1) may not be present (consider initramfs)
/dev may not contain /dev/console or /dev/tty, which POSIX mandates

Where in libstd does it call out to the basename binary or open /dev/console? We're talking about a compiled program and not a shell environment. You're still discussing apples to oranges.

Contributor

cardoe commented May 10, 2016

@eternaleye

...the fact that various things required by POSIX can be absent during early boot?

Mandatory utilities such as basename(1) may not be present (consider initramfs)
/dev may not contain /dev/console or /dev/tty, which POSIX mandates

Where in libstd does it call out to the basename binary or open /dev/console? We're talking about a compiled program and not a shell environment. You're still discussing apples to oranges.

@eternaleye

This comment has been minimized.

Show comment
Hide comment
@eternaleye

eternaleye May 10, 2016

Contributor

I never said it did either. I said early boot is not a POSIX environment, and gave two examples I was able to find quickly, rather than spend hours wading through the spec for more targeted ones.

Contributor

eternaleye commented May 10, 2016

I never said it did either. I said early boot is not a POSIX environment, and gave two examples I was able to find quickly, rather than spend hours wading through the spec for more targeted ones.

@tbu-

This comment has been minimized.

Show comment
Hide comment
@tbu-

tbu- May 10, 2016

Contributor

@cardoe I think that you're overly aggressive about this.

Contributor

tbu- commented May 10, 2016

@cardoe I think that you're overly aggressive about this.

@cardoe cardoe deleted the cardoe:non-blocking-rand-read branch Jun 22, 2016

@stouset

This comment has been minimized.

Show comment
Hide comment
@stouset

stouset Jul 7, 2016

Sorry if I'm late to the party — I came here after reading the patch notes for 1.10 and being concerned about the notion about reverting to /dev/urandom if getrandom(2) blocks.

It seems to me this is a poor security tradeoff. Rust programs that are intended for early-boot environments are surely a somewhat-rare use case. In these cases, such binaries should be forced to opt-in to reduced security — in some cases it's probably not a big deal, but for others it can and will cause security issues.

This isn't a small project that will barely see the light of day — as Rust maintainers, you're making security decisions and tradeoffs that affect all of your users. And trading away strong security by default to favor rare use cases (early-boot programs that are short-lived and/or have bounded datasets in HashMaps) does a disservice to the majority.

stouset commented Jul 7, 2016

Sorry if I'm late to the party — I came here after reading the patch notes for 1.10 and being concerned about the notion about reverting to /dev/urandom if getrandom(2) blocks.

It seems to me this is a poor security tradeoff. Rust programs that are intended for early-boot environments are surely a somewhat-rare use case. In these cases, such binaries should be forced to opt-in to reduced security — in some cases it's probably not a big deal, but for others it can and will cause security issues.

This isn't a small project that will barely see the light of day — as Rust maintainers, you're making security decisions and tradeoffs that affect all of your users. And trading away strong security by default to favor rare use cases (early-boot programs that are short-lived and/or have bounded datasets in HashMaps) does a disservice to the majority.

@brson

This comment has been minimized.

Show comment
Hide comment
@brson

brson Jul 7, 2016

Contributor

@stouset I suggest filing a bug about it. Comments here will be ineffective.

I'm also a bit worried about this, but it's important to remember that the only thing this affects is the order things are inserted into hash tables. That is, the only attack this could enable is hash table DOS. It's not a weakening of random numbers in Rust in general.

Contributor

brson commented Jul 7, 2016

@stouset I suggest filing a bug about it. Comments here will be ineffective.

I'm also a bit worried about this, but it's important to remember that the only thing this affects is the order things are inserted into hash tables. That is, the only attack this could enable is hash table DOS. It's not a weakening of random numbers in Rust in general.

@stouset

This comment has been minimized.

Show comment
Hide comment
@stouset

stouset Jul 8, 2016

Yes, I realize that's the case. This is, to be clear, a somewhat rare use-case that the overwhelming majority Rust developers will likely never encounter.

stouset commented Jul 8, 2016

Yes, I realize that's the case. This is, to be clear, a somewhat rare use-case that the overwhelming majority Rust developers will likely never encounter.

jonathandturner added a commit to jonathandturner/rust that referenced this pull request Aug 8, 2016

Rollup merge of #35371 - mgattozzi:master, r=steveklabnik
Update HashMap docs regarding DoS protection

Because of changes to how Rust acquires randomness HashMap is not
guaranteed to be DoS resistant. This commit reflects these changes in
the docs themselves and provides an alternative method to creating
a hash that is resistant if needed.

This fixes #33817 and includes relevant information regarding changes made in #33086

steveklabnik added a commit to steveklabnik/rust that referenced this pull request Aug 10, 2016

Rollup merge of #35371 - mgattozzi:master, r=steveklabnik
Update HashMap docs regarding DoS protection

Because of changes to how Rust acquires randomness HashMap is not
guaranteed to be DoS resistant. This commit reflects these changes in
the docs themselves and provides an alternative method to creating
a hash that is resistant if needed.

This fixes #33817 and includes relevant information regarding changes made in #33086
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment