An in-memory RNG that shares its file descriptor. #14351

asajeffrey · 2016-11-23T22:25:51Z

This PR implements an in-memory random number generator that only uses an OS RNG for (re)seeding. The OS RNG is shared, so there's only one file descriptor for /dev/urandom being used.

The PR also implements a tidy check that we don't accidentally introduce an RNG. Rather annoyingly, there are a lot of transitive dependencies on rand, notably hash maps in std.

This PR makes it possible to use uuids for identifiers such as pipeline and frame ids.

./mach build -d does not report any errors
./mach test-tidy does not report any errors
These changes do not require tests because it's fixing a resource issue

This change is

highfive · 2016-11-23T22:25:55Z

Heads up! This PR modifies the following files:

@bholley: components/style/Cargo.toml
@wafflespeanut: python/tidy/servo_tidy/tidy.py
@KiChjang: components/script/lib.rs, components/script/Cargo.toml, components/script/dom/crypto.rs, components/script/dom/htmlformelement.rs, components/script/dom/dedicatedworkerglobalscope.rs, components/script/dom/serviceworkerglobalscope.rs
@fitzgen: components/script/lib.rs, components/script/Cargo.toml, components/script/dom/crypto.rs, components/script/dom/htmlformelement.rs, components/script/dom/dedicatedworkerglobalscope.rs, components/script/dom/serviceworkerglobalscope.rs
@emilio: components/style/Cargo.toml

highfive · 2016-11-23T22:25:56Z

Warning

These commits modify unsafe code. Please review it carefully!
These commits modify style and script code, but no tests are modified. Please consider adding a test!

asajeffrey · 2016-11-23T22:26:35Z

cc @ConnorGBrewster @larsbergstrom @nox

asajeffrey · 2016-11-23T22:29:36Z

@avadacatavra this replaces the RNG in crypto. We should double-check that we are okay with that.

emilio · 2016-11-23T22:33:48Z

FWIW, rayon uses rand but only for XorShiftRng, so probably we should be able to filter this in a better way?

asajeffrey · 2016-11-23T22:40:08Z

We don't scan dependency source, so it's difficult to see how we can do much better than a whitelist.

nox · 2016-11-24T10:32:30Z

That's quite a lot of code. I would rather we use the Fnv one explicitly when we see failures due to file limits with a RNG-related stack, than use a custom hasher.

asajeffrey · 2016-11-24T15:28:45Z

@nox: yes, we can use Fnv for hash maps, but that doesn't help for uuids or other cases where we need an RNG, not a hash function.

Most of the code here is just cut-and-paste from rand, in particular there's no new RNG algorithms, it just reuses existing ones. The only new bit is sharing an OsRng (and hence sharing an fd) rather than creating a new one for each thread.

bors-servo · 2016-11-28T12:06:10Z

☔ The latest upstream changes (presumably #14381) made this pull request unmergeable. Please resolve the merge conflicts.

bors-servo · 2016-11-29T21:14:12Z

☔ The latest upstream changes (presumably #14292) made this pull request unmergeable. Please resolve the merge conflicts.

bors-servo · 2016-12-07T03:27:08Z

☔ The latest upstream changes (presumably #14473) made this pull request unmergeable. Please resolve the merge conflicts.

bors-servo · 2016-12-15T01:41:21Z

☔ The latest upstream changes (presumably #14592) made this pull request unmergeable. Please resolve the merge conflicts.

asajeffrey · 2016-12-15T01:57:53Z

Rebased.

bors-servo · 2016-12-21T13:11:44Z

☔ The latest upstream changes (presumably #14652) made this pull request unmergeable. Please resolve the merge conflicts.

asajeffrey · 2016-12-23T14:56:43Z

Rebased. @emilio or @nox: any chance of a review?

bors-servo · 2016-12-24T13:47:19Z

☔ The latest upstream changes (presumably #14687) made this pull request unmergeable. Please resolve the merge conflicts.

emilio · 2016-12-25T22:24:58Z

This is fine for me when rebased, but maybe @nox has still concerns.

wafflespeanut · 2017-01-05T13:27:48Z

python/tidy/servo_tidy/tidy.py

@@ -81,6 +81,24 @@
    " accessible to\n// web pages."
 ]

+WHITELISTED_DEPENDENCIES = {


We should really be moving this to the config file, but that shouldn't be a concern for this PR. I'll take care of it in an easy issue 😄

asajeffrey · 2017-01-05T13:43:39Z

IRC conversation with @nox: http://logs.glob.uno/?c=mozilla%23servo&s=5+Jan+2017&e=5+Jan+2017#c587629

TL;DR he r+s once it's split into two commits, and WHITELISTED_DEPENDENCIES is renamed.


        An in-memory RNG that shares its file descriptor.


        Added test-tidy check for blocked packages.

asajeffrey · 2017-01-05T13:57:36Z

@bors-servo r=emilio

bors-servo · 2017-01-05T13:57:37Z

📌 Commit e3a8e3b has been approved by emilio

bors-servo · 2017-01-05T14:32:25Z

⌛ Testing commit e3a8e3b with merge 36ddf76...


        Auto merge of #14351 - asajeffrey:servo-rand-share-fds, r=emilio

An in-memory RNG that shares its file descriptor.  This PR implements an in-memory random number generator that only uses an OS RNG for (re)seeding. The OS RNG is shared, so there's only one file descriptor for `/dev/urandom` being used. The PR also implements a tidy check that we don't accidentally introduce an RNG. Rather annoyingly, there are a lot of transitive dependencies on `rand`, notably hash maps in `std`. This PR makes it possible to use uuids for identifiers such as pipeline and frame ids. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes do not require tests because it's fixing a resource issue   --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14351)

bors-servo · 2017-01-05T15:33:42Z

☀️ Test successful - android, arm32, arm64, linux-dev, linux-rel-css, linux-rel-wpt, mac-dev-unit, mac-rel-css, mac-rel-wpt1, mac-rel-wpt2, windows-gnu-dev, windows-msvc-dev

dhardy · 2018-01-14T14:02:57Z

@asajeffrey can you tell me what the motivation for this is?

I'm working on improving rand, and I'd like to know if there is a deficiency. It may be that OsRng can use multiple file descriptors, but if so we should fix it internally.

jdm · 2018-01-14T14:52:56Z

The motivation was the frequency with which Service would run out of file descriptors during tests. I don't recall seeing that problem since making this change

jdm · 2018-01-14T14:53:14Z

Servo, not Service.

asajeffrey · 2018-01-14T16:24:17Z

@dhardy, sharing the OS RNG among threads, to avoid running out of FDs, but only using the shared RNG for reseeding, to avoid lock contention.

asajeffrey added the I-perf-bloat label Nov 23, 2016

highfive assigned emilio Nov 23, 2016

highfive added the S-awaiting-review label Nov 23, 2016

highfive added the S-needs-rebase label Nov 28, 2016

asajeffrey force-pushed the asajeffrey:servo-rand-share-fds branch from ae888f8 to 5f79e29 Nov 28, 2016

asajeffrey removed the S-needs-rebase label Nov 28, 2016

highfive added the S-needs-rebase label Nov 29, 2016

asajeffrey force-pushed the asajeffrey:servo-rand-share-fds branch from 5f79e29 to a25da00 Dec 3, 2016

asajeffrey removed the S-needs-rebase label Dec 3, 2016

highfive added the S-needs-rebase label Dec 7, 2016

asajeffrey force-pushed the asajeffrey:servo-rand-share-fds branch from a25da00 to 91ce41f Dec 8, 2016

asajeffrey removed the S-needs-rebase label Dec 8, 2016

asajeffrey mentioned this pull request Dec 13, 2016

Use unguessable PipelineIds and FrameIds #14283

Closed

4 of 5 tasks complete

highfive added the S-needs-rebase label Dec 15, 2016

asajeffrey force-pushed the asajeffrey:servo-rand-share-fds branch 2 times, most recently from dc05dda to 1026d4d Dec 15, 2016

asajeffrey removed the S-needs-rebase label Dec 15, 2016

highfive added the S-needs-rebase label Dec 21, 2016

asajeffrey force-pushed the asajeffrey:servo-rand-share-fds branch from 1026d4d to f2248e4 Dec 23, 2016

highfive added the S-needs-rebase label Dec 24, 2016

wafflespeanut reviewed Jan 5, 2017

View changes

asajeffrey force-pushed the asajeffrey:servo-rand-share-fds branch from f2248e4 to 70ffe1f Jan 5, 2017

asajeffrey added 2 commits Jan 5, 2017

An in-memory RNG that shares its file descriptor.

7ace30f

Added test-tidy check for blocked packages.

Loading status checks…

e3a8e3b

asajeffrey force-pushed the asajeffrey:servo-rand-share-fds branch from 70ffe1f to e3a8e3b Jan 5, 2017

highfive added S-awaiting-merge and removed S-awaiting-review S-needs-rebase labels Jan 5, 2017

bors-servo merged commit e3a8e3b into servo:master Jan 5, 2017
2 of 3 checks passed

2 of 3 checks passed

continuous-integration/travis-ci/pr The Travis CI build failed
Details

continuous-integration/appveyor/pr AppVeyor build succeeded
Details

homu Test successful
Details

highfive removed the S-awaiting-merge label Jan 5, 2017

wafflespeanut mentioned this pull request Jan 13, 2017

Move tidy's `BLOCKED_PACKAGES` list to its config #15014

Closed

servo / servo

An in-memory RNG that shares its file descriptor. #14351

An in-memory RNG that shares its file descriptor. #14351

asajeffrey commented Nov 23, 2016 •

edited

highfive commented Nov 23, 2016

highfive commented Nov 23, 2016

asajeffrey commented Nov 23, 2016

asajeffrey commented Nov 23, 2016

emilio commented Nov 23, 2016

asajeffrey commented Nov 23, 2016

nox commented Nov 24, 2016

asajeffrey commented Nov 24, 2016

bors-servo commented Nov 28, 2016

bors-servo commented Nov 29, 2016

bors-servo commented Dec 7, 2016

bors-servo commented Dec 15, 2016

asajeffrey commented Dec 15, 2016

bors-servo commented Dec 21, 2016

asajeffrey commented Dec 23, 2016

bors-servo commented Dec 24, 2016

emilio commented Dec 25, 2016

This comment has been minimized.

asajeffrey commented Jan 5, 2017

asajeffrey commented Jan 5, 2017

bors-servo commented Jan 5, 2017

bors-servo commented Jan 5, 2017

bors-servo commented Jan 5, 2017

This comment has been minimized.

dhardy commented on `7ace30f` Jan 14, 2018

This comment has been minimized.

jdm replied Jan 14, 2018

This comment has been minimized.

jdm replied Jan 14, 2018

This comment has been minimized.

asajeffrey replied Jan 14, 2018

servo / servo

Join GitHub today

An in-memory RNG that shares its file descriptor. #14351

An in-memory RNG that shares its file descriptor. #14351

Conversation

asajeffrey commented Nov 23, 2016 • edited

highfive commented Nov 23, 2016

highfive commented Nov 23, 2016

asajeffrey commented Nov 23, 2016

asajeffrey commented Nov 23, 2016

emilio commented Nov 23, 2016

asajeffrey commented Nov 23, 2016

nox commented Nov 24, 2016

asajeffrey commented Nov 24, 2016

bors-servo commented Nov 28, 2016

bors-servo commented Nov 29, 2016

bors-servo commented Dec 7, 2016

bors-servo commented Dec 15, 2016

asajeffrey commented Dec 15, 2016

bors-servo commented Dec 21, 2016

asajeffrey commented Dec 23, 2016

bors-servo commented Dec 24, 2016

emilio commented Dec 25, 2016

This comment has been minimized.

wafflespeanut Jan 5, 2017

asajeffrey commented Jan 5, 2017

asajeffrey commented Jan 5, 2017

bors-servo commented Jan 5, 2017

bors-servo commented Jan 5, 2017

bors-servo commented Jan 5, 2017

This comment has been minimized.

dhardy commented on 7ace30f Jan 14, 2018

This comment has been minimized.

jdm replied Jan 14, 2018

This comment has been minimized.

jdm replied Jan 14, 2018

This comment has been minimized.

asajeffrey replied Jan 14, 2018

asajeffrey commented Nov 23, 2016 •

edited

dhardy commented on `7ace30f` Jan 14, 2018