New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use `HybridBitSet` in `SparseBitMatrix`. #54318

Merged
merged 3 commits into from Sep 19, 2018

Conversation

Projects
None yet
10 participants
@nnethercote
Contributor

nnethercote commented Sep 18, 2018

This fixes most of the remaining NLL memory regression.

r? @pnkfelix, because you reviewed #54286.
cc @nikomatsakis, because NLL
cc @Mark-Simulacrum, because this removes array_vec.rs
cc @lqd, because this massively improves unic-ucd-name, and probably other public crates

nnethercote added some commits Sep 18, 2018

Remove `array_vec.rs`.
`SparseBitSet` is the only remaining user of `ArrayVec`. This commit
switches it to using `SmallVec`, and removes `array_vec.rs`.

Why the switch? Although `SparseBitSet` is size-limited and doesn't need
the ability to spill to the heap, `SmallVec` has many more features than
`ArrayVec`. In particular, it's now possible to keep `SparseBitSet`'s
elements in sorted order, which gives in-order iteration, which is a
requirement for the next commit.
@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Sep 18, 2018

⚠️ Warning ⚠️

  • These commits modify submodules.
@rust-highfive

This comment was marked as outdated.

Collaborator

rust-highfive commented Sep 18, 2018

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
#####################################                                     52.1%
######################################################################## 100.0%
[00:02:02] extracting /checkout/obj/build/cache/2018-09-11/cargo-beta-x86_64-unknown-linux-gnu.tar.gz
[00:02:04]     Updating registry `https://github.com/rust-lang/crates.io-index`
[00:02:28] error: the lock file needs to be updated but --locked was passed to prevent this
[00:02:28] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:28] Makefile:81: recipe for target 'prepare' failed
[00:02:28] make: *** [prepare] Error 1
[00:02:29] Command failed. Attempt 2/5:
[00:02:29] Command failed. Attempt 2/5:
[00:02:31] error: the lock file needs to be updated but --locked was passed to prevent this
[00:02:31] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:31] make: *** [prepare] Error 1
[00:02:31] Makefile:81: recipe for target 'prepare' failed
[00:02:33] Command failed. Attempt 3/5:
[00:02:33] Command failed. Attempt 3/5:
[00:02:33] error: the lock file needs to be updated but --locked was passed to prevent this
[00:02:33] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:33] make: *** [prepare] Error 1
[00:02:33] Makefile:81: recipe for target 'prepare' failed
[00:02:36] Command failed. Attempt 4/5:
[00:02:36] Command failed. Attempt 4/5:
[00:02:36] error: the lock file needs to be updated but --locked was passed to prevent this
[00:02:36] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:36] Makefile:81: recipe for target 'prepare' failed
[00:02:36] make: *** [prepare] Error 1
[00:02:40] Command failed. Attempt 5/5:
[00:02:40] Command failed. Attempt 5/5:
[00:02:41] error: the lock file needs to be updated but --locked was passed to prevent this
[00:02:41] failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
[00:02:41] Makefile:81: recipe for target 'prepare' failed
[00:02:41] make: *** [prepare] Error 1
[00:02:41] The command has failed after 5 attempts.
travis_time:end:2bf35670:start=1537252588752718227,finish=1537252755206919928,duration=166454201701
---
travis_time:end:0463602c:start=1537252755615275698,finish=1537252755622805334,duration=7529636
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0b4deb16
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:2ed0a696
travis_time:start:2ed0a696
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:00f01a33
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Use `HybridBitSet` for rows within `SparseBitMatrix`.
This requires adding a few extra methods to `HybridBitSet`. (These are
tested in a new unit test.)

This commit reduces the `max-rss` for `nll-check` builds of `html5ever`
by 46%, `ucd` by 45%, `clap-rs` by 23%, `inflate` by 14%. And the
results for the `unic-ucd-name` crate are even more impressive: a 21%
reduction in instructions, a 60% reduction in wall-time, a 96%
reduction in `max-rss`, and a 97% reduction in faults!

Fixes #52028.
@nnethercote

This comment has been minimized.

Contributor

nnethercote commented Sep 18, 2018

The submodules changes were accidental, and I have reverted them.

@nnethercote

This comment has been minimized.

Contributor

nnethercote commented Sep 18, 2018

[00:02:28] error: the lock file needs to be updated but --locked was passed to prevent this

The second commit is probably the cause of this, because it removes array_vec.rs... but I don't know what needs to be done.

@nnethercote

This comment has been minimized.

Contributor

nnethercote commented Sep 18, 2018

@bors try

@bors

This comment has been minimized.

Contributor

bors commented Sep 18, 2018

⌛️ Trying commit 154be2c with merge c243f30...

bors added a commit that referenced this pull request Sep 18, 2018

Auto merge of #54318 - nnethercote:use-HybridBitSet-in-SparseBitMatri…
…x, r=<try>

Use `HybridBitSet` in `SparseBitMatrix`.

This fixes most of the remaining NLL memory regression.

r? @pnkfelix, because you reviewed #54286.
cc @nikomatsakis, because NLL
cc @Mark-Simulacrum, because this removes `array_vec.rs`
cc @lqd, because this massively improves `unic-ucd-name`, and probably other public crates
@bors

This comment has been minimized.

Contributor

bors commented Sep 18, 2018

☀️ Test successful - status-travis
State: approved= try=True

@nnethercote

This comment has been minimized.

Contributor

nnethercote commented Sep 18, 2018

@rust-timer

This comment has been minimized.

rust-timer commented Sep 18, 2018

Success: Queued c243f30 with parent b80cb47, comparison URL.

@pnkfelix

This comment has been minimized.

Member

pnkfelix commented Sep 18, 2018

Looks great to me.

The dummy bitsets are unfortunate, but I'd rather land this first and then after that, maybe I will investigate revising the coding patterns to get rid of them.

@pnkfelix

This comment has been minimized.

Member

pnkfelix commented Sep 18, 2018

@bors r+

@bors

This comment has been minimized.

Contributor

bors commented Sep 18, 2018

📌 Commit 154be2c has been approved by pnkfelix

@pnkfelix

This comment has been minimized.

Member

pnkfelix commented Sep 18, 2018

ah, I guess it would be prudent to wait until the results are back from rust-timer.

@bors r-

@nnethercote

This comment has been minimized.

Contributor

nnethercote commented Sep 18, 2018

I expect the rust-timer results will be much the same as what I got locally, but sure, we can wait.

@lqd

This comment has been minimized.

Contributor

lqd commented Sep 18, 2018

perf results are ready, awesome improvements for max-rss as expected 🎉

@pnkfelix

This comment has been minimized.

Member

pnkfelix commented Sep 18, 2018

@bors r+

@bors

This comment has been minimized.

Contributor

bors commented Sep 18, 2018

📌 Commit 154be2c has been approved by pnkfelix

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Sep 18, 2018

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
✓ uploaded script
travis_fold:end:step_upload_script
travis_fold:start:worker_info
Worker information
hostname: 8693e843-d64a-4db0-a622-2d8a987f7042@1.production-2-worker-org-gce-zm1x
instance: travis-job-a76ee0cc-3f8b-45e9-afbf-a3b90ac37f86 travis-ci-connie-trusty-1512502258-986baf0 (via amqp)
startup: 48.856560272s
travis_fold:end:worker_info
travis_fold:start:system_info
---
Attempting to download s3://rust-lang-ci-sccache2/docker/fb846323fae811c75bb1af129bb81e3ccd01260054711cf61af1fd9e28020555f48a094c896bb167411b9470d64a9b0660359d4933a1576f31626975541a2dc0
[00:19:40] Attempting with retry: curl -f -L -C - -o /tmp/rustci_docker_cache https://s3-us-west-1.amazonaws.com/rust-lang-ci-sccache2/docker/fb846323fae811c75bb1af129bb81e3ccd01260054711cf61af1fd9e28020555f48a094c896bb167411b9470d64a9b0660359d4933a1576f31626975541a2dc0
[00:19:40]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[00:19:40]                                  Dload  Upload   Total   Spent    Left  Speed
No output has been received in the last 30m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@pietroalbini

This comment has been minimized.

Member

pietroalbini commented Sep 18, 2018

@bors retry #40474

@bors

This comment has been minimized.

Contributor

bors commented Sep 18, 2018

⌛️ Testing commit 154be2c with merge 64c407e...

bors added a commit that referenced this pull request Sep 18, 2018

Auto merge of #54318 - nnethercote:use-HybridBitSet-in-SparseBitMatri…
…x, r=pnkfelix

Use `HybridBitSet` in `SparseBitMatrix`.

This fixes most of the remaining NLL memory regression.

r? @pnkfelix, because you reviewed #54286.
cc @nikomatsakis, because NLL
cc @Mark-Simulacrum, because this removes `array_vec.rs`
cc @lqd, because this massively improves `unic-ucd-name`, and probably other public crates

@memoryruins memoryruins added the A-NLL label Sep 18, 2018

@bors

This comment has been minimized.

Contributor

bors commented Sep 18, 2018

💔 Test failed - status-travis

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Sep 18, 2018

The job dist-x86_64-linux-alt of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
✓ uploaded script
travis_fold:end:step_upload_script
travis_fold:start:worker_info
Worker information
hostname: 7b607e1c-491c-44fc-9630-8fbd518d037d@1.production-2-worker-org-gce-zfk4
instance: travis-job-a5986c39-ca1a-4f1c-816d-0ee5983b21f7 travis-ci-connie-trusty-1512502258-986baf0 (via amqp)
startup: 27.59767184s
travis_fold:end:worker_info
travis_fold:start:system_info
---
[02:56:25]    Compiling clippy_lints v0.0.212 (file:///checkout/src/tools/clippy/clippy_lints)
[02:56:26]    Compiling cargo_metadata v0.6.0
[02:56:38]    Compiling url v1.7.1
[02:58:05] [RUSTC-TIMING] clippy_lints test:false 100.281
The job exceeded the maximum time limit for jobs, and has been terminated.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@shepmaster

This comment has been minimized.

Member

shepmaster commented Sep 18, 2018

@bors retry

@bors

This comment has been minimized.

Contributor

bors commented Sep 18, 2018

⌛️ Testing commit 154be2c with merge 2e51a80...

bors added a commit that referenced this pull request Sep 18, 2018

Auto merge of #54318 - nnethercote:use-HybridBitSet-in-SparseBitMatri…
…x, r=pnkfelix

Use `HybridBitSet` in `SparseBitMatrix`.

This fixes most of the remaining NLL memory regression.

r? @pnkfelix, because you reviewed #54286.
cc @nikomatsakis, because NLL
cc @Mark-Simulacrum, because this removes `array_vec.rs`
cc @lqd, because this massively improves `unic-ucd-name`, and probably other public crates
@nnethercote

This comment has been minimized.

Contributor

nnethercote commented Sep 18, 2018

The dummy bitsets are unfortunate, but I'd rather land this first and then after that, maybe I will investigate revising the coding patterns to get rid of them.

I'd love to know how to avoid them. I asked about this for a similar example in /r/rust and the answers all involved repeated matches and unreachable!()s.

@bors

This comment has been minimized.

Contributor

bors commented Sep 19, 2018

💔 Test failed - status-appveyor

@nnethercote

This comment has been minimized.

Contributor

nnethercote commented Sep 19, 2018

AFAICT, the i686-pc-windows-gnu machine hit the 3 hr limit. (The x86_64-pc-windows-gnu build took 2h58m.)

@bors retry, I guess

@nnethercote

This comment has been minimized.

Contributor

nnethercote commented Sep 19, 2018

@bors retry

@bors

This comment has been minimized.

Contributor

bors commented Sep 19, 2018

⌛️ Testing commit 154be2c with merge ff6422d...

bors added a commit that referenced this pull request Sep 19, 2018

Auto merge of #54318 - nnethercote:use-HybridBitSet-in-SparseBitMatri…
…x, r=pnkfelix

Use `HybridBitSet` in `SparseBitMatrix`.

This fixes most of the remaining NLL memory regression.

r? @pnkfelix, because you reviewed #54286.
cc @nikomatsakis, because NLL
cc @Mark-Simulacrum, because this removes `array_vec.rs`
cc @lqd, because this massively improves `unic-ucd-name`, and probably other public crates
@bors

This comment has been minimized.

Contributor

bors commented Sep 19, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: pnkfelix
Pushing ff6422d to master...

@bors bors merged commit 154be2c into rust-lang:master Sep 19, 2018

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details

@nnethercote nnethercote deleted the nnethercote:use-HybridBitSet-in-SparseBitMatrix branch Sep 19, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment