New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace HashMap implementation with SwissTable #56241

Open
wants to merge 19 commits into
base: master
from

Conversation

@Amanieu
Contributor

Amanieu commented Nov 26, 2018

The implementation is from the hashbrown crate.

This is mostly complete, however it is missing 2 features:

  • Adaptive early resizing. -- doesn't seem to be needed
  • try_reserve -- DONE

cc @pietroalbini @Gankro

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Nov 26, 2018

r? @sfackler

(rust_highfive has picked a reviewer for you, use r? to override)

@pietroalbini

This comment has been minimized.

Member

pietroalbini commented Nov 26, 2018

@bors try

cc @rust-lang/infra, this should get a perf run as soon as the try build finishes (it's needed for the second day of impl days at rustfest rome)

bors added a commit that referenced this pull request Nov 26, 2018

Auto merge of #56241 - Amanieu:hashbrown2, r=<try>
WIP: Replace HashMap implementation with SwissTable

The implementation is from the [hashbrown](https://github.com/Amanieu/hashbrown) crate.

This is mostly complete, however it is missing 2 features:
- Adaptive early resizing.
- `try_reserve`

cc @pietroalbini @Gankro
@bors

This comment has been minimized.

Contributor

bors commented Nov 26, 2018

⌛️ Trying commit cfd3225 with merge 5aaebb9...

@rust-highfive

This comment was marked as outdated.

Collaborator

rust-highfive commented Nov 26, 2018

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:1529e080:start=1543249255992989136,finish=1543249321851190115,duration=65858200979
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
[00:03:36]    Compiling panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
[00:03:40] warning: unused variable: `additional`
[00:03:40]    --> src/libstd/collections/hash/map.rs:412:35
[00:03:40]     |
[00:03:40] 412 |     pub fn try_reserve(&mut self, additional: usize) -> Result<(), CollectionAllocErr> {
[00:03:40]     |                                   ^^^^^^^^^^ help: consider using `_additional` instead
[00:03:40]     = note: #[warn(unused_variables)] on by default
[00:03:40] 
[00:03:52]     Finished release [optimized] target(s) in 45.76s
[00:03:52] Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
[00:18:31]    Compiling core v0.0.0 (/checkout/src/libcore)
[00:18:31]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:18:31]    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
[00:18:36] error[E0432]: unresolved import
[00:18:36]   --> src/libcore/num/dec2flt/rawfp.rs:33:11
[00:18:36]    |
[00:18:36] 33 | use fmt::{Debug, LowerExp};
[00:18:36] 
[00:18:36] error: cannot determine resolution for the derive macro `Debug`
[00:18:36] error: cannot determine resolution for the derive macro `Debug`
[00:18:36]   --> src/libcore/num/dec2flt/rawfp.rs:40:23
[00:18:36]    |
[00:18:36] 40 | #[derive(Copy, Clone, Debug)]
[00:18:36]    |
[00:18:36]    = note: import resolution is stuck, try simplifying macro imports
[00:18:36] 
[00:18:36] 
[00:18:37] error[E0416]: identifier `Acquire` is bound more than once in the same pattern
[00:18:37]      |
[00:18:37]      |
[00:18:37] 2091 |         (Acquire, Acquire) => intrinsics::atomic_cxchg_acq(dst, old, new),
[00:18:37]      |                   ^^^^^^^ used in a pattern more than once
[00:18:37] 
[00:18:37] error[E0416]: identifier `Relaxed` is bound more than once in the same pattern
[00:18:37]      |
[00:18:37]      |
[00:18:37] 2094 |         (Relaxed, Relaxed) => intrinsics::atomic_cxchg_relaxed(dst, old, new),
[00:18:37]      |                   ^^^^^^^ used in a pattern more than once
[00:18:37] 
[00:18:37] error[E0416]: identifier `SeqCst` is bound more than once in the same pattern
[00:18:37]      |
[00:18:37]      |
[00:18:37] 2095 |         (SeqCst, SeqCst) => intrinsics::atomic_cxchg(dst, old, new),
[00:18:37]      |                  ^^^^^^ used in a pattern more than once
[00:18:37] 
[00:18:37] error[E0416]: identifier `Acquire` is bound more than once in the same pattern
[00:18:37]      |
[00:18:37]      |
[00:18:37] 2116 |         (Acquire, Acquire) => intrinsics::atomic_cxchgweak_acq(dst, old, new),
[00:18:37]      |                   ^^^^^^^ used in a pattern more than once
[00:18:37] 
[00:18:37] error[E0416]: identifier `Relaxed` is bound more than once in the same pattern
[00:18:37]      |
[00:18:37]      |
[00:18:37] 2119 |         (Relaxed, Relaxed) => intrinsics::atomic_cxchgweak_relaxed(dst, old, new),
[00:18:37]      |                   ^^^^^^^ used in a pattern more than once
[00:18:37] 
[00:18:37] error[E0416]: identifier `SeqCst` is bound more than once in the same pattern
[00:18:37]      |
[00:18:37]      |
[00:18:37] 2120 |         (SeqCst, SeqCst) => intrinsics::atomic_cxchgweak(dst, old, new),
[00:18:37]      |                  ^^^^^^ used in a pattern more than once
[00:18:37] error: aborting due to 8 previous errors
[00:18:37] 
[00:18:37] Some errors occurred: E0416, E0432.
[00:18:37] For more information about an error, try `rustc --explain E0416`.
[00:18:37] For more information about an error, try `rustc --explain E0416`.
[00:18:37] error: Could not compile `core`.
[00:18:37] warning: build failed, waiting for other jobs to finish...
[00:18:38] error: build failed
[00:18:38] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" "panic-unwind backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "--message-format" "json"
[00:18:38] expected success, got: exit code: 101
[00:18:38] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap build
[00:18:38] Build completed unsuccessfully in 0:15:32
[00:18:38] Makefile:28: recipe for target 'all' failed
[00:18:38] make: *** [all] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:04a839fb
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Mon Nov 26 16:40:49 UTC 2018

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@pietroalbini

This comment has been minimized.

Member

pietroalbini commented Nov 26, 2018

@bors try delegate+

@bors

This comment has been minimized.

Contributor

bors commented Nov 26, 2018

✌️ @Amanieu can now approve this pull request

@bors

This comment has been minimized.

Contributor

bors commented Nov 26, 2018

⌛️ Trying commit 03db6f4 with merge 73919a1...

bors added a commit that referenced this pull request Nov 26, 2018

Auto merge of #56241 - Amanieu:hashbrown2, r=<try>
WIP: Replace HashMap implementation with SwissTable

The implementation is from the [hashbrown](https://github.com/Amanieu/hashbrown) crate.

This is mostly complete, however it is missing 2 features:
- Adaptive early resizing.
- `try_reserve`

cc @pietroalbini @Gankro
@rust-highfive

This comment was marked as outdated.

Collaborator

rust-highfive commented Nov 26, 2018

The job dist-x86_64-linux of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:1f8f5510
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
[00:05:22]    Compiling panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
[00:05:26] warning: unused variable: `additional`
[00:05:26]    --> src/libstd/collections/hash/map.rs:412:35
[00:05:26]     |
[00:05:26] 412 |     pub fn try_reserve(&mut self, additional: usize) -> Result<(), CollectionAllocErr> {
[00:05:26]     |                                   ^^^^^^^^^^ help: consider using `_additional` instead
[00:05:26]     = note: #[warn(unused_variables)] on by default
[00:05:26] 
[00:05:40]     Finished release [optimized] target(s) in 50.48s
[00:05:40] Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
[00:49:56]    Compiling core v0.0.0 (/checkout/src/libcore)
[00:49:56]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:49:56]    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
[00:50:00] error[E0432]: unresolved import
[00:50:00]   --> src/libcore/num/dec2flt/rawfp.rs:33:11
[00:50:00]    |
[00:50:00] 33 | use fmt::{Debug, LowerExp};
[00:50:00] 
[00:50:00] error: cannot determine resolution for the derive macro `Debug`
[00:50:00] error: cannot determine resolution for the derive macro `Debug`
[00:50:00]   --> src/libcore/num/dec2flt/rawfp.rs:40:23
[00:50:00]    |
[00:50:00] 40 | #[derive(Copy, Clone, Debug)]
[00:50:00]    |
[00:50:00]    = note: import resolution is stuck, try simplifying macro imports
[00:50:00] 
[00:50:00] 
[00:50:01] error[E0416]: identifier `Acquire` is bound more than once in the same pattern
[00:50:01]      |
[00:50:01]      |
[00:50:01] 2091 |         (Acquire, Acquire) => intrinsics::atomic_cxchg_acq(dst, old, new),
[00:50:01]      |                   ^^^^^^^ used in a pattern more than once
[00:50:01] 
[00:50:01] error[E0416]: identifier `Relaxed` is bound more than once in the same pattern
[00:50:01]      |
[00:50:01]      |
[00:50:01] 2094 |         (Relaxed, Relaxed) => intrinsics::atomic_cxchg_relaxed(dst, old, new),
[00:50:01]      |                   ^^^^^^^ used in a pattern more than once
[00:50:01] 
[00:50:01] error[E0416]: identifier `SeqCst` is bound more than once in the same pattern
[00:50:01]      |
[00:50:01]      |
[00:50:01] 2095 |         (SeqCst, SeqCst) => intrinsics::atomic_cxchg(dst, old, new),
[00:50:01]      |                  ^^^^^^ used in a pattern more than once
[00:50:01] 
[00:50:01] error[E0416]: identifier `Acquire` is bound more than once in the same pattern
[00:50:01]      |
[00:50:01]      |
[00:50:01] 2116 |         (Acquire, Acquire) => intrinsics::atomic_cxchgweak_acq(dst, old, new),
[00:50:01]      |                   ^^^^^^^ used in a pattern more than once
[00:50:01] 
[00:50:01] error[E0416]: identifier `Relaxed` is bound more than once in the same pattern
[00:50:01]      |
[00:50:01]      |
[00:50:01] 2119 |         (Relaxed, Relaxed) => intrinsics::atomic_cxchgweak_relaxed(dst, old, new),
[00:50:01]      |                   ^^^^^^^ used in a pattern more than once
[00:50:01] 
[00:50:01] error[E0416]: identifier `SeqCst` is bound more than once in the same pattern
[00:50:01]      |
[00:50:01]      |
[00:50:01] 2120 |         (SeqCst, SeqCst) => intrinsics::atomic_cxchgweak(dst, old, new),
[00:50:01]      |                  ^^^^^^ used in a pattern more than once
[00:50:02] error: aborting due to 8 previous errors
[00:50:02] 
[00:50:02] Some errors occurred: E0416, E0432.
[00:50:02] For more information about an error, try `rustc --explain E0416`.
---
travis_time:end:0d9aa0f5:start=1543252380560909682,finish=1543252380570961872,duration=10052190
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:009923ec
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:2056231c
travis_time:start:2056231c
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:14508654
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rust-highfive

This comment was marked as outdated.

Collaborator

rust-highfive commented Nov 26, 2018

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:03b58678:start=1543251899547419502,finish=1543251953974008641,duration=54426589139
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
[00:19:12]    Compiling core v0.0.0 (/checkout/src/libcore)
[00:19:12]    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
[00:19:12]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:19:17] error[E0432]: unresolved import
[00:19:17]   --> src/libcore/num/dec2flt/rawfp.rs:33:11
[00:19:17]    |
[00:19:17] 33 | use fmt::{Debug, LowerExp};
[00:19:17] 
[00:19:17] error: cannot determine resolution for the derive macro `Debug`
[00:19:17] error: cannot determine resolution for the derive macro `Debug`
[00:19:17]   --> src/libcore/num/dec2flt/rawfp.rs:40:23
[00:19:17]    |
[00:19:17] 40 | #[derive(Copy, Clone, Debug)]
[00:19:17]    |
[00:19:17]    = note: import resolution is stuck, try simplifying macro imports
[00:19:17] 
[00:19:17] 
[00:19:17] error[E0416]: identifier `Acquire` is bound more than once in the same pattern
[00:19:17]      |
[00:19:17]      |
[00:19:17] 2091 |         (Acquire, Acquire) => intrinsics::atomic_cxchg_acq(dst, old, new),
[00:19:17]      |                   ^^^^^^^ used in a pattern more than once
[00:19:17] 
[00:19:17] error[E0416]: identifier `Relaxed` is bound more than once in the same pattern
[00:19:17]      |
[00:19:17]      |
[00:19:17] 2094 |         (Relaxed, Relaxed) => intrinsics::atomic_cxchg_relaxed(dst, old, new),
[00:19:17]      |                   ^^^^^^^ used in a pattern more than once
[00:19:17] 
[00:19:17] error[E0416]: identifier `SeqCst` is bound more than once in the same pattern
[00:19:17]      |
[00:19:17]      |
[00:19:17] 2095 |         (SeqCst, SeqCst) => intrinsics::atomic_cxchg(dst, old, new),
[00:19:17]      |                  ^^^^^^ used in a pattern more than once
[00:19:17] 
[00:19:17] error[E0416]: identifier `Acquire` is bound more than once in the same pattern
[00:19:17]      |
[00:19:17]      |
[00:19:17] 2116 |         (Acquire, Acquire) => intrinsics::atomic_cxchgweak_acq(dst, old, new),
[00:19:17]      |                   ^^^^^^^ used in a pattern more than once
[00:19:17] 
[00:19:17] error[E0416]: identifier `Relaxed` is bound more than once in the same pattern
[00:19:17]      |
[00:19:17]      |
[00:19:17] 2119 |         (Relaxed, Relaxed) => intrinsics::atomic_cxchgweak_relaxed(dst, old, new),
[00:19:17]      |                   ^^^^^^^ used in a pattern more than once
[00:19:17] 
[00:19:17] error[E0416]: identifier `SeqCst` is bound more than once in the same pattern
[00:19:17]      |
[00:19:17] 2120error: aborting due to 8 previous errors
[00:19:19] 
[00:19:19] Some errors occurred: E0416, E0432.
---
[00:19:19] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" "panic-unwind backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "--message-format" "json"
[00:19:19] expected success, got: exit code: 101
[00:19:19] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap build
[00:19:19] Build completed unsuccessfully in 0:15:50
[00:19:19] Makefile:28: recipe for target 'all' failed
[00:19:19] make: *** [all] Error 1
57304 ./obj/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib
57300 ./obj/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu
57296 ./obj/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib
56896 ./src/llvm/test/MC

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors

This comment has been minimized.

Contributor

bors commented Nov 26, 2018

💔 Test failed - status-travis

@rust-highfive

This comment was marked as outdated.

Collaborator

rust-highfive commented Nov 26, 2018

The job dist-x86_64-linux of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:05edbc7a
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
[00:47:07]    Compiling core v0.0.0 (/checkout/src/libcore)
[00:47:07]    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
[00:47:07]    Compiling unwind v0.0.0 (/checkout/src/libunwind)
[00:47:10] error[E0432]: unresolved import
[00:47:10]   --> src/libcore/num/dec2flt/rawfp.rs:33:11
[00:47:10]    |
[00:47:10] 33 | use fmt::{Debug, LowerExp};
[00:47:10] 
[00:47:10] error: cannot determine resolution for the derive macro `Debug`
[00:47:10] error: cannot determine resolution for the derive macro `Debug`
[00:47:10]   --> src/libcore/num/dec2flt/rawfp.rs:40:23
[00:47:10]    |
[00:47:10] 40 | #[derive(Copy, Clone, Debug)]
[00:47:10]    |
[00:47:10]    = note: import resolution is stuck, try simplifying macro imports
[00:47:10] 
[00:47:10] 
[00:47:10] error[E0416]: identifier `Acquire` is bound more than once in the same pattern
[00:47:10]      |
[00:47:10]      |
[00:47:10] 2091 |         (Acquire, Acquire) => intrinsics::atomic_cxchg_acq(dst, old, new),
[00:47:10]      |                   ^^^^^^^ used in a pattern more than once
[00:47:10] 
[00:47:10] error[E0416]: identifier `Relaxed` is bound more than once in the same pattern
[00:47:10]      |
[00:47:10]      |
[00:47:10] 2094 |         (Relaxed, Relaxed) => intrinsics::atomic_cxchg_relaxed(dst, old, new),
[00:47:10]      |                   ^^^^^^^ used in a pattern more than once
[00:47:10] 
[00:47:10] error[E0416]: identifier `SeqCst` is bound more than once in the same pattern
[00:47:10]      |
[00:47:10]      |
[00:47:10] 2095 |         (SeqCst, SeqCst) => intrinsics::atomic_cxchg(dst, old, new),
[00:47:10]      |                  ^^^^^^ used in a pattern more than once
[00:47:10] 
[00:47:10] error[E0416]: identifier `Acquire` is bound more than once in the same pattern
[00:47:10]      |
[00:47:10]      |
[00:47:10] 2116 |         (Acquire, Acquire) => intrinsics::atomic_cxchgweak_acq(dst, old, new),
[00:47:10]      |                   ^^^^^^^ used in a pattern more than once
[00:47:10] 
[00:47:10] error[E0416]: identifier `Relaxed` is bound more than once in the same pattern
[00:47:10]      |
[00:47:10]      |
[00:47:10] 2119 |         (Relaxed, Relaxed) => intrinsics::atomic_cxchgweak_relaxed(dst, old, new),
[00:47:10]      |                   ^^^^^^^ used in a pattern more than once
[00:47:10] 
[00:47:10] error[E0416]: identifier `SeqCst` is bound more than once in the same pattern
[00:47:10]      |
[00:47:10]      |
[00:47:10] 2120 |         (SeqCst, SeqCst) => intrinsics::atomic_cxchgweak(dst, old, new),
[00:47:10]      |                  ^^^^^^ used in a pattern more than once
[00:47:11] error: aborting due to 8 previous errors
[00:47:11] 
[00:47:11] Some errors occurred: E0416, E0432.
[00:47:11] For more information about an error, try `rustc --explain E0416`.
---
travis_time:end:0a9aeb56:start=1543254768289710638,finish=1543254768295832212,duration=6121574
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:056e60b6
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1c0c9ede
travis_time:start:1c0c9ede
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:171ff89d
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Amanieu

This comment has been minimized.

Contributor

Amanieu commented Nov 27, 2018

It seems that the new HashMap is causing errors in name resolution. However I am not familiar with the code. Could someone point me at the HashMaps that are likely to cause the errors above?

cc @petrochenkov

@petrochenkov

This comment has been minimized.

Contributor

petrochenkov commented Nov 27, 2018

@Amanieu
use fmt::Debug blocks resolution of #[derive(Debug)].
Apparently we can't determine that it doesn't import anything named Debug in macro namespace soon enough.

The workaround is to remove use fmt::Debug; and use fmt::Debug in code.

@petrochenkov

This comment has been minimized.

Contributor

petrochenkov commented Nov 27, 2018

Oh, wait a second, this PR doesn't actually modify rawfp.rs and libcore in general.
That's worse.

@petrochenkov

This comment has been minimized.

Contributor

petrochenkov commented Nov 27, 2018

Did the previous HashMap have any guarantees about "order of elements == order of insertion", or something like that, accidentally?

@Amanieu

This comment has been minimized.

Contributor

Amanieu commented Nov 27, 2018

@petrochenkov There are no ordering guarantees in HashMap, so it is very likely that the ordering has changed when the HashMap implementation changed. Is there anything in name resolution that relies on the insertion order?

I am currently trying to narrow down the exact hash map that is causing the issue. I can confirm that switching only librustc_resolve to the new hashmap reproduces this issue (everything else still uses the old hashmap).

Note that it is also possible that there is a bug in the new HashMap.

@petrochenkov

This comment has been minimized.

Contributor

petrochenkov commented Nov 27, 2018

There are no ordering guarantees in HashMap, so it is very likely that the ordering has changed

Fix: order of elements with a hash collision, not the general order.
But that probably doesn't matter because our hash maps cannot be used as multi-maps.

I am currently trying to narrow down the exact hash map that is causing the issue. I can confirm that switching only librustc_resolve to the new hashmap reproduces this issue (everything else still uses the old hashmap).

Yes, it's better to bisect all uses of FxHashMap in rustc_resolve and find out what exact map is responsible, then it will be clearer what happens on the resolve side.

@Gankro

This comment has been minimized.

Contributor

Gankro commented Dec 5, 2018

Is it basically accurate to say this implementation is a "normal" quadratic probing hashmap with a cute trick (the side-index of partial hashes and metadata) to improve cache effects? Which is to say, we could cite any random paper describing the properties of a hashmap with quadratic probing and tombstones? Modulo an adjustment to the 50% capacity limit due to grouping 16 elements into a linear group before doing a jump?

@Amanieu

This comment has been minimized.

Contributor

Amanieu commented Dec 5, 2018

@Gankro Pretty much. The actual algorithm is independent of the the group size (some platforms use 4/8 element groups instead), so I presume that an analysis with a group size of 1 would still apply (i.e. the group size can be ignored as a constant factor).

@bors

This comment has been minimized.

Contributor

bors commented Dec 7, 2018

☔️ The latest upstream changes (presumably #56066) made this pull request unmergeable. Please resolve the merge conflicts.

@Gankro

This comment has been minimized.

Contributor

Gankro commented Dec 7, 2018

Has anyone actually reviewed the details of the implementation? (...are people just expecting me to?)

@Gankro

This comment has been minimized.

Contributor

Gankro commented Dec 10, 2018

Ok starting a review now, probably gonna take a while.

NB: I am just reviewing all of the code from scratch since this is enough of a rewrite that anything else seems error-prone and annoying. So you might get some random comments on stuff you didn't write. Do with that what you will. Also gonna have lots of random questions and pedantry just because I want to be sure I'm not missing things.

@Gankro

Ok, here's the first batch: reviewing the bitmask/group modules (mostly nit picks as I wrap my head around the code).

Next up, reviewing Raw and Map.

Show resolved Hide resolved src/libstd/collections/hash/raw/bitmask.rs
Show resolved Hide resolved src/libstd/collections/hash/raw/bitmask.rs Outdated
Show resolved Hide resolved src/libstd/collections/hash/raw/bitmask.rs Outdated
Show resolved Hide resolved src/libstd/collections/hash/raw/bitmask.rs Outdated
pub type BitMaskWord = GroupWord;
pub const BITMASK_SHIFT: u32 = 3;
pub const BITMASK_MASK: GroupWord = 0x8080808080808080u64 as GroupWord;

This comment has been minimized.

@Gankro

Gankro Dec 10, 2018

Contributor
Suggested change Beta
pub const BITMASK_MASK: GroupWord = 0x8080808080808080u64 as GroupWord;
// Set the high bit of every byte
pub const BITMASK_MASK: BitMaskWord = 0x8080808080808080u64 as BitMaskWord;

(minimizing hexpertese required, and also BitMaskWord seems vaguely more correct here)

This comment has been minimized.

@Gankro

Gankro Dec 10, 2018

Contributor

alternatively: can we make repeat a const fn and just use it here? I guess this would be hostile towards the out-of-tree impl?

Show resolved Hide resolved src/libstd/collections/hash/raw/generic.rs
Show resolved Hide resolved src/libstd/collections/hash/raw/generic.rs
Show resolved Hide resolved src/libstd/collections/hash/raw/generic.rs
Show resolved Hide resolved src/libstd/collections/hash/raw/generic.rs
Show resolved Hide resolved src/libstd/collections/hash/raw/sse2.rs Outdated
@Gankro

checkpoint in raw.rs as Amanieu is pushing a change

use core::ptr::NonNull;
// Extracted from the scopeguard crate
mod scopeguard {

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

So this is just a finally block for exception-safety?

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

Also personal preference: I like to chuck vendored/utility junk like this at the bottom, as I want to start with the key/important stuff first (along with a big comment explaining the high level design).

}
// Branch prediction hint. This is currently only available on nightly but it
// consistently improves performance by 10-15%.

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

this is a weird comment to have in-tree but I assume this is for the benefit of the out-of-tree impl.

This comment has been minimized.

@Amanieu

Amanieu Dec 12, 2018

Contributor

It's also an inaccurate one, and I think I will end up removing the likely/unlikely annotations considering that they have negative performance impact now.

/// Primary hash function, used to select the initial bucket to probe from.
#[inline]
fn h1(hash: u64) -> usize {

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

I know the h1/h2 scheme is from the cpp impl, but I wonder if better names aren't possible? low_bits/high_bits? to_index/to_entry..? (I mostly don't like h1/h2 because it's easy to mix them up)

}
/// Probe sequence based on triangular numbers, which is guaranteed (since our
/// table size is a power of two) to visit every group of elements exactly once.

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

[citation needed]

}
/// Probe sequence based on triangular numbers, which is guaranteed (since our
/// table size is a power of two) to visit every group of elements exactly once.

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor
Suggested change Beta
/// table size is a power of two) to visit every group of elements exactly once.
/// table size is a power of two) to visit every group of elements exactly once.
/// A triangular probe has us jump by 1 more group every time. So first we
/// jump by 1 group (meaning we just continue our linear scan), then 2 groups
/// (skipping over 1 group), then 3 groups (skipping over 2 groups), and so on.
struct ProbeSeq {
mask: usize,
offset: usize,
index: usize,

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

naming nits: I feel like index would be better named something like probe_stride or jump_stride. Meanwhile offset really just is the index into the table's arrays, right? It's not relative or scaled or anything? (mask might also benefit from appealing to the affect that it's the capacity and/or used for modulo. modulo_mask? Or at least be more consistent and use bucket_mask as used elsewhere.

} else {
((bucket_mask + 1) / 8) * 7
}
}

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

NB: * 8 / 7 / 8 * 7 isn't the identity for integers but that's "fine" since we don't guarantee capacity/load-factor be that precise. Although ideally Map::with_capacity(x).capacity() >= x. Due to the next_power_of_two, this actually always turns out to be true! (experimentally confirmed)

Here is a proof that, for all x >= 0:

x <= ((x * 8 / 7).floor().next_power_of_two() / 8).floor() * 7

Remember 8 is a power of two, so it always divides a larger power of two cleanly, producing another power of two. So the second floor is a noop. Capacities for this table are therefore always of the form 7 * 2^k, and in fact the distance between x and capacity is minimized and exactly zero when x = 7 * 2^k:

  ((7 * 2^k * 8 / 7).floor().next_power_of_two() / 8).floor() * 7
= ((2^(k + 3) * 7 / 7).floor().next_power_of_two() / 8).floor() * 7  # rearrange *8
= ((2^(k + 3)).floor().next_power_of_two() / 8).floor() * 7          # * 7 / 7 cancels
= ((2^(k + 3)).next_power_of_two() / 8).floor() * 7                  # flooring int does nothing
= (2^(k + 3) / 8).floor() * 7                                        # 2^(k+3) is a power of two 
= (2^k).floor() * 7                                                  # collapse /8 into power
= 7 * 2^k                                                            # flooring integer does nothing

If we increase x at all, such as with x = 7 * 2^k + 1, then we jump to the next power of two:

   (((7 * 2^k + 1) * 8 / 7).floor().next_power_of_two() / 8).floor() * 7
 = ((7 * 2^(k + 3) + 8) / 7).floor().next_power_of_two() / 8).floor() * 7  # distribute *8
 = ((2^(k + 3) + 8 / 7).floor().next_power_of_two() / 8).floor() * 7       # distribute /7
 = ((2^(k + 3) + 1).next_power_of_two() / 8).floor() * 7                   # flooring turns +8/7 into +1
 = ((2^(k + 4) / 8).floor() * 7                                            # next_power turns +1 into *2
 = (2^(k + 1)).floor() * 7                                                 # collapse /8 into power
 = 7 * 2^(k + 1)                                                           # flooring int does nothing
>= 7 * 2^k + 1, for all k >= 0

Therefore Map::with_capacity(x).capacity() >= x for all inputs (since we catch overflow and such)

#[inline]
pub unsafe fn drop(&self) {
self.ptr.as_ptr().drop_in_place();
}

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

A comment explaining why these are all &self would be appreciated...

This comment has been minimized.

@Amanieu

Amanieu Dec 12, 2018

Contributor

... I have no good reason for that. The methods will still be unsafe, since Bucket is not tied to the lifetime of the underlying table.

data: NonNull<T>,
items: usize,
growth_left: usize,
}

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

Interesting, the c++ impl doesn't toss all this metadata in the allocation?

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

Also can you comment what these fields are?

///
/// In effect this returns a table with exactly 1 bucket. However we can
/// leave the data pointer dangling since that bucket is never written to
/// due to our load factor forcing us to always have at least 1 free bucket.

This comment has been minimized.

@Gankro

Gankro Dec 11, 2018

Contributor

[at this point in reading the code linearly] unclear why get/remove won't dive in here

This comment has been minimized.

@Amanieu

Amanieu Dec 12, 2018

Contributor

Because of load factor, a table with 1 bucket has a capacity of 0. This means that the table will be resized before insertion. And remove will never reach this since the table is always empty so there is nothing to remove.

@Amanieu Amanieu force-pushed the Amanieu:hashbrown2 branch from f770977 to ec35c14 Dec 11, 2018

@Amanieu Amanieu force-pushed the Amanieu:hashbrown2 branch from ec35c14 to 3bee3c0 Dec 11, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment