WIP: add raw_entry API to HashMap #50821

Gankro · May 17, 2018

This is an implementation of https://internals.rust-lang.org/t/pre-rfc-abandonning-morals-in-the-name-of-performance-the-raw-entry-api/7043 with some minor tweaks.

TODO:

write docs (nothing super interesting except for the top-level)
write Debug impls
create tracking issue
audit naming
port more code to use raw_entry instead of entry?
remove the K: Eq requirement from raw_entry? (currently exists to satisfy internal APIs, but I don't think it's strictly needed)
write tests

rust-highfive · May 17, 2018

r? @withoutboats

(rust_highfive has picked a reviewer for you, use r? to override)

withoutboats · May 17, 2018

@Gankro pick a reviewer who is competent pls

Gankro · May 17, 2018

The actual impl is ~trivial since everything that was needed is already in hashmap for implementation reasons. So this mostly just needs API review from the @rust-lang/libs team.

Gankro · May 17, 2018

I'm a bit worried removing the K: Eq requirement will mess up the quality of the rustdoc output

Gankro · May 17, 2018

Oh. Not requiring K: Hash would also effectively mandate that HashMap stores the full hashes at all times. This isn't great, as at very least it would be worth considering truncating the hashes for "small" (read: ~every) HashMap for perf reasons.

pietroalbini · May 21, 2018

r? @alexcrichton

alexcrichton · May 25, 2018

alexcrichton reviewed May 25, 2018

View changes

Thanks and sorry for the delay @Gankro! Since these are all starting as unstable it's ok to not get a full libs-team sign-off because we'll do that before stabilization anyway. I do think that we'll want these in the long run so it seems good to add them to libstd to start experimenting with them.

I'll admit though that I haven't followed the RFC too too closely so this was the first time I was looking at a number of these APIs. I was expecting something to be unsafe due to the "raw" terminology but it ended up not being so! I was then a little perplexed how this was sort of a standalone API from the existing Entry API, but I think that's because Entry basically implies an owned key stored internally, right?

alexcrichton · May 25, 2018

src/libstd/collections/hash/map.rs

+    ///
+    /// Immutable raw entries have very limited use; you might instead want `raw_entry`.
+    #[unstable(feature = "raw_entry", issue = "42069")]
+    pub fn raw_entry_immut(&self) -> RawImmutableEntryBuilder<K, V, S> {


Naming-wise could this perhaps be raw_entry and the one above be raw_entry_mut?

Yeah I'm split on it

raw_entry/raw_entry_mut is absolutely more idiomatic, but entry matches raw_entry_mut and raw_entry_mut is the really important one.

I'm definitely being "weird" here, and am willing to relent if anyone feels strong about it.

alexcrichton · May 25, 2018

src/libstd/collections/hash/map.rs

+    /// assert_eq!(map["poneyland"], 22);
+    /// ```
+    #[unstable(feature = "raw_entry", issue = "42069")]
+    pub fn or_insert(self, default_key: K, default_val: V) -> (&'a mut K, &'a mut V) {


FWIW Entry::or_insert only returns &mut V so this is somewhat inconsistent with that, albeit more general

that's definitely an intentional design decision, but I could be convinced to go back

alexcrichton · May 25, 2018

src/libstd/collections/hash/map.rs

+    /// use std::collections::hash_map::Entry;
+    ///
+    /// let mut map: HashMap<&str, u32> = HashMap::new();
+    /// map.entry("poneyland").or_insert(12);


A number of these doc strings I've noticed are using entry, but I think they may want to move towards raw_entry?

fixed, this was just copy-pasted junk i hadn't updated yet

rust-highfive · May 25, 2018

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:04:04]    Compiling std_unicode v0.0.0 (file:///checkout/src/libstd_unicode)
[00:04:06]    Compiling alloc_system v0.0.0 (file:///checkout/src/liballoc_system)
[00:04:06]    Compiling panic_abort v0.0.0 (file:///checkout/src/libpanic_abort)
[00:04:12]    Compiling panic_unwind v0.0.0 (file:///checkout/src/libpanic_unwind)
[00:04:15] error[E0277]: `K` doesn't implement `core::fmt::Debug`
[00:04:15]      |
[00:04:15]      |
[00:04:15] 2278 |          .field("key", self.key())
[00:04:15]      |                        ^^^^^^^^^^ `K` cannot be formatted using `{:?}` because it doesn't implement `core::fmt::Debug`
[00:04:15]      |
[00:04:15]      = help: the trait `core::fmt::Debug` is not implemented for `K`
[00:04:15]      = help: consider adding a `where K: core::fmt::Debug` bound
[00:04:15]      = note: required for the cast to the object type `core::fmt::Debug`
[00:04:15] 
[00:04:15] error[E0277]: `V` doesn't implement `core::fmt::Debug`
[00:04:15]      |
[00:04:15]      |
[00:04:15] 2279 |          .field("value", self.get())
[00:04:15]      |                          ^^^^^^^^^^ `V` cannot be formatted using `{:?}` because it doesn't implement `core::fmt::Debug`
[00:04:15]      |
[00:04:15]      = help: the trait `core::fmt::Debug` is not implemented for `V`
[00:04:15]      = help: consider adding a `where V: core::fmt::Debug` bound
[00:04:15]      = note: required for the cast to the object type `core::fmt::Debug`
[00:04:17] error: aborting due to 2 previous errors
[00:04:17] 
[00:04:17] For more information about this error, try `rustc --explain E0277`.
[00:04:17] error: Could not compile `std`.
[00:04:17] error: Could not compile `std`.
[00:04:17] 
[00:04:17] Caused by:
[00:04:17]   process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name std libstd/lib.rs --color always --error-format json --crate-type dylib --crate-type rlib --emit=dep-info,link -C prefer-dynamic -C opt-level=3 --cfg feature="alloc_jemalloc" --cfg feature="backtrace" --cfg feature="jemalloc" --cfg feature="panic-unwind" --cfg feature="panic_unwind" -C metadata=1eeccaae4c437516 -C extra-filename=-1eeccaae4c437516 --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/release/deps --extern panic_unwind=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libpanic_unwind-472f8a98a21071c0.rlib --extern rustc_tsan=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/librustc_tsan-933c25e53a64afa3.rlib --extern unwind=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libunwind-de93819b3358210b.rlib --extern alloc_system=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liballoc_system-d6c788271c3d4cb5.rlib --extern panic_abort=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libpanic_abort-7033dc785bf77e76.rlib --extern libc=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liblibc-904ce106c1301515.rlib --extern compiler_builtins=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcompiler_builtins-48ff328ae5e9f1b4.rlib --extern rustc_lsan=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/librustc_lsan-71a0b5c1c68b7c91.rlib --extern alloc_jemalloc=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liballoc_jemalloc-d5e800cf2beb81d3.rlib --extern rustc_msan=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/librustc_msan-ecf5e4d4cc2515ca.rlib --extern rustc_asan=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/librustc_asan-469443ee7ab8941b.rlib --extern alloc=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liballoc-0e25c4bddec1c94e.rlib --extern core=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcore-0d1ebef792b1d9ca.rlib --extern std_unicode=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libstd_unicode-7be48b1ecbc2ee28.rlib -L native=/checkout/obj/build/x86_64-unknown-linux-gnu/native/libbacktrace/.libs -l static=backtrace -l dl -l rt -l pthread -L native=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/build/compiler_builtins-7a2aed77e6522e53/out -L native=/checkout/obj/build/x86_64-unknown-linux-gnu/native/jemalloc/lib` (exit code: 101)
[00:04:17] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "--message-format" "json"
[00:04:17] expected success, got: exit code: 101
[00:04:17] thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1091:9
[00:04:17] travis_fold:end:stage0-std

[00:04:17] travis_time:end:stage0-std:start=1527274980881865755,finish=1527275036595429583,duration=55713563828


[00:04:17] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:17] Build completed unsuccessfully in 0:00:57
[00:04:17] Makefile:79: recipe for target 'tidy' failed
[00:04:17] make: *** [tidy] Error 1

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0ab7dda2
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

rust-highfive · May 25, 2018

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:47:46] .........................................................i..........................................
[00:47:51] .............................................................................ii.....................
[00:47:57] ....................................................................................................
[00:48:03] .......................................................................................i............
[00:48:05] .....iiiiiiiii...................................................
[00:48:05] 
[00:48:05] travis_fold:start:test_ui_nll
travis_time:start:test_ui_nll
Check compiletest suite=ui mode=ui compare_mode=nll (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
[00:48:53] .........................................................i..........................................
[00:48:58] .............................................................................ii.....................
[00:49:03] ....................................................................................................
[00:49:08] .......................................................................................i............
[00:49:11] .....iiiiiiiii...................................................
[00:49:11] 
[00:49:11]  finished in 65.386
[00:49:11] travis_fold:end:test_ui_nll

---
[01:17:14] travis_fold:start:test_stage1-std
travis_time:start:test_stage1-std
Testing std stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:17:15]    Compiling std v0.0.0 (file:///checkout/src/libstd)
[01:17:21] error[E0689]: can't call method `hash` on ambiguous numeric type `{integer}`
[01:17:21]      |
[01:17:21]      |
[01:17:21] 4266 |             1.hash(&mut h);
[01:17:21]      |               ^^^^
[01:17:21] help: you must specify a concrete type for this numeric value, like `i32`
[01:17:21]      |
[01:17:21] 4266 |             1_i32.hash(&mut h);
[01:17:21] 
[01:17:21] 
[01:17:21] error[E0599]: no method named `finish` found for type `collections::hash::map::DefaultHasher` in the current scope
[01:17:21]      |
[01:17:21]      |
[01:17:21] 3194 | pub struct DefaultHasher(SipHasher13);
[01:17:21]      | -------------------------------------- method `finish` not found for this
[01:17:21] ...
[01:17:21] 4267 |             h.finish()
[01:17:21]      |
[01:17:21]      = help: items from traits can only be used if the trait is in scope
[01:17:21]      = note: the following trait is implemented but not in scope, perhaps add a `use` for it:
[01:17:21]              candidate #1: `use core::hash::Hasher;`
[01:17:21]              candidate #1: `use core::hash::Hasher;`
[01:17:21] 
[01:17:21] error[E0689]: can't call method `hash` on ambiguous numeric type `{integer}`
[01:17:21]      |
[01:17:21]      |
[01:17:21] 4287 |             3.hash(&mut h);
[01:17:21]      |               ^^^^
[01:17:21] help: you must specify a concrete type for this numeric value, like `i32`
[01:17:21]      |
[01:17:21] 4287 |             3_i32.hash(&mut h);
[01:17:21] 
[01:17:21] 
[01:17:21] error[E0599]: no method named `finish` found for type `collections::hash::map::DefaultHasher` in the current scope
[01:17:21]      |
[01:17:21]      |
[01:17:21] 3194 | pub struct DefaultHasher(SipHasher13);
[01:17:21]      | -------------------------------------- method `finish` not found for this
[01:17:21] ...
[01:17:21] 4288 |             h.finish()
[01:17:21]      |
[01:17:21]      = help: items from traits can only be used if the trait is in scope
[01:17:21]      = note: the following trait is implemented but not in scope, perhaps add a `use` for it:
[01:17:21]              candidate #1: `use core::hash::Hasher;`
[01:17:21]              candidate #1: `use core::hash::Hasher;`
[01:17:21] 
[01:17:21] error[E0599]: no method named `remove_kv` found for type `collections::hash::map::RawOccupiedEntry<'_, {integer}, {integer}>` in the current scope
[01:17:21]      |
[01:17:21]      |
[01:17:21] 1866 | pub struct RawOccupiedEntry<'a, K: 'a, V: 'a> {
[01:17:21]      | --------------------------------------------- method `remove_kv` not found for this
[01:17:21] ...
[01:17:21] 4292 |                 assert_eq!(view.remove_kv(), (3, 30));
[01:17:21]      |
[01:17:21]      |
[01:17:21]      = help: did you mean `remove`?
[01:17:30] error: aborting due to 5 previous errors
[01:17:30] 
[01:17:30] Some errors occurred: E0599, E0689.
[01:17:30] For more information about an error, try `rustc --explain E0599`.
[01:17:30] For more information about an error, try `rustc --explain E0599`.
[01:17:30] error: Could not compile `std`.
[01:17:30] 
[01:17:30] To learn more, run the command again with --verbose.
[01:17:30] 
[01:17:30] 
[01:17:30] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "-p" "std" "--" "--quiet"
[01:17:30] 
[01:17:30] 
[01:17:30] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:17:30] Build completed unsuccessfully in 0:32:02
[01:17:30] Build completed unsuccessfully in 0:32:02
[01:17:30] Makefile:58: recipe for target 'check' failed
[01:17:30] make: *** [check] Error 1

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:32bee10a
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

rust-highfive · May 26, 2018

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

Requirement already satisfied: PyYAML<=3.12,>=3.10 in /usr/lib/python2.7/dist-packages (from awscli)
Collecting botocore==1.10.28 (from awscli)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/2e/91/f0870d4de8eb78897ce781f3ff22fc492bbb9849b5c91f26db20b125ef36/botocore-1.10.28-py2.py3-none-any.whl (4.2MB)
    0% |                                | 10kB 39.5MB/s eta 0:00:01
    0% |▏                               | 20kB 41.4MB/s eta 0:00:01
    0% |▎                               | 30kB 36.1MB/s eta 0:00:01
    0% |▎                               | 40kB 17.5MB/s eta 0:00:01
---
[00:44:27] .........................................................i..........................................
[00:44:31] ..............................................................................ii....................
[00:44:37] ....................................................................................................
[00:44:43] .......................................................................................i............
[00:44:45] .....iiiiiiiii...................................................
[00:44:45] 
[00:44:45] travis_fold:start:test_ui_nll
travis_time:start:test_ui_nll
Check compiletest suite=ui mode=ui compare_mode=nll (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
[00:45:32] .........................................................i..........................................
[00:45:36] .............................................................................ii.....................
[00:45:41] ....................................................................................................
[00:45:47] .......................................................................................i............
[00:45:49] .....iiiiiiiii...................................................
[00:45:49] 
[00:45:49]  finished in 63.515
[00:45:49] travis_fold:end:test_ui_nll

---
[01:11:52] travis_fold:start:test_stage1-std
travis_time:start:test_stage1-std
Testing std stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:11:52]    Compiling std v0.0.0 (file:///checkout/src/libstd)
[01:11:58] error[E0599]: no method named `hash` found for type `i32` in the current scope
[01:11:58]      |
[01:11:58]      |
[01:11:58] 4266 |             1i32.hash(&mut h);
[01:11:58]      |
[01:11:58]      = help: items from traits can only be used if the trait is in scope
[01:11:58]      = note: the following trait is implemented but not in scope, perhaps add a `use` for it:
[01:11:58]              candidate #1: `use core::hash::Hash;`
[01:11:58]              candidate #1: `use core::hash::Hash;`
[01:11:58] 
[01:11:58] error[E0599]: no method named `finish` found for type `collections::hash::map::DefaultHasher` in the current scope
[01:11:58]      |
[01:11:58]      |
[01:11:58] 3194 | pub struct DefaultHasher(SipHasher13);
[01:11:58]      | -------------------------------------- method `finish` not found for this
[01:11:58] ...
[01:11:58] 4267 |             h.finish()
[01:11:58]      |
[01:11:58]      = help: items from traits can only be used if the trait is in scope
[01:11:58]      = note: the following trait is implemented but not in scope, perhaps add a `use` for it:
[01:11:58]              candidate #1: `use core::hash::Hasher;`
[01:11:58]              candidate #1: `use core::hash::Hasher;`
[01:11:58] 
[01:11:58] error[E0599]: no method named `hash` found for type `i32` in the current scope
[01:11:58]      |
[01:11:58]      |
[01:11:58] 4287 |             3i32.hash(&mut h);
[01:11:58]      |
[01:11:58]      = help: items from traits can only be used if the trait is in scope
[01:11:58]      = note: the following trait is implemented but not in scope, perhaps add a `use` for it:
[01:11:58]              candidate #1: `use core::hash::Hash;`
[01:11:58]              candidate #1: `use core::hash::Hash;`
[01:11:58] 
[01:11:58] error[E0599]: no method named `finish` found for type `collections::hash::map::DefaultHasher` in the current scope
[01:11:58]      |
[01:11:58]      |
[01:11:58] 3194 | pub struct DefaultHasher(SipHasher13);
[01:11:58]      | -------------------------------------- method `finish` not found for this
[01:11:58] ...
[01:11:58] 4288 |             h.finish()
[01:11:58]      |
[01:11:58]      = help: items from traits can only be used if the trait is in scope
[01:11:58]      = note: the following trait is implemented but not in scope, perhaps add a `use` for it:
[01:11:58]              candidate #1: `use core::hash::Hasher;`
[01:11:58]              candidate #1: `use core::hash::Hasher;`
[01:11:58] 
[01:11:58] error[E0599]: no method named `remove_kv` found for type `collections::hash::map::RawOccupiedEntry<'_, i32, i32>` in the current scope
[01:11:58]      |
[01:11:58]      |
[01:11:58] 1866 | pub struct RawOccupiedEntry<'a, K: 'a, V: 'a> {
[01:11:58]      | --------------------------------------------- method `remove_kv` not found for this
[01:11:58] ...
[01:11:58] 4292 |                 assert_eq!(view.remove_kv(), (3, 30));
[01:11:58]      |
[01:11:58]      |
[01:11:58]      = help: did you mean `remove`?
[01:12:06] error: aborting due to 5 previous errors
[01:12:06] 
[01:12:06] For more information about this error, try `rustc --explain E0599`.
[01:12:06] error: Could not compile `std`.
[01:12:06] error: Could not compile `std`.
[01:12:06] 
[01:12:06] To learn more, run the command again with --verbose.
[01:12:06] 
[01:12:06] 
[01:12:06] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "-p" "std" "--" "--quiet"
[01:12:06] 
[01:12:06] 
[01:12:06] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:12:06] Build completed unsuccessfully in 0:29:50
[01:12:06] Build completed unsuccessfully in 0:29:50
[01:12:06] make: *** [check] Error 1
[01:12:06] Makefile:58: recipe for target 'check' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:19abce3b
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

shepmaster · Jun 2, 2018

Ping from triage, @Gankro ! You got some test failures and had a question from your reviewer.

TimNN · Jun 12, 2018

@Gankro: We haven't heard from you in two weeks, so we're closing this PR for now. Feel free to re-open in the future.

Gankro · Jun 14, 2018

will pick this up again in a bit, just had a productive meeting with @aturon:

go to standard x, x_mut naming
make the immutable case return a proper enum to be consistent with BTreeMap, and also to do conveniences
otherwise try to hew close to Entry for consistency

Amanieu · Jul 3, 2018

@Gankro Are you still working on this? I am interested in taking over if you are too busy to continue working on it.

fintelia · Jul 3, 2018

fintelia reviewed Jul 3, 2018

View changes

fintelia · Jul 3, 2018

src/libstd/collections/hash/map.rs

+    /// In particular, the hash used to initialized the raw entry must still be
+    /// consistent with the hash of the key that is ultimately stored in the entry.
+    /// This is because implementations of HashMap may need to recompute hashes
+    /// when resizing, at which point only the keys are available.


It is worth pointing out that this specific case isn't actually a concern for the current implementation because it always stores the full hash. Rather the main issue is the failure case described below in which the entry is in the wrong place and thus becomes "lost".

While this is true for the current implementation, this might change in the future and code should not rely on that.

We should make sure that this section doesn't come across as saying that bogus hashes are OK as long as the HashMap isn't resized.

Actually, reading that paragraph again, it isn't clear to me why the case described would be a problem: if the HashMap is current in an invalid state due to bogus hashes then rehashing everything would fix it, not break it further

Consider a type Foo(u32, u32), who's Hash impl is just Foo.0.hash(), and you use this API to actually feed in Foo.1.hash().

If resizing triggers a rehash, then your code will work perfectly reasonably until a resize triggers, at which point everything will be moved to the location indicated by Foo.0, while you're still performing lookups with Foo.1. All existing keys will appear to "vanish" from the map.

Ah, I see what you are saying. If you exclusively use the raw entry API with a bogus hash function things will work until elements are rehashed using the real hash function. Using the normal get/insert/entry functions would be have the opposite problem of working only once the resize happened.

Gankro · Jul 5, 2018

@Amanieu please do!

Gankro · Jul 5, 2018

Also here is an example of usage: https://gist.github.com/Gankro/fb0bfe6f6770aba09b9a1cdf0ecf47e0

fintelia · Jul 5, 2018

fintelia reviewed Jul 5, 2018

View changes

fintelia · Jul 5, 2018

src/libstd/collections/hash/map.rs

+        mem::replace(self.get_mut(), value)
+    }
+
+    /// Sets the value of the entry, and returns the entry's old value.


nit: value -> key

fintelia · Jul 5, 2018

How would people feel extending this API to provide a RawEntryBuilder::from_index() function that produced a raw entry directly from an index in the HashMap? This would make it possible to treat a HashMap as an array of raw entries (which is an accurate view of how it is implemented) and would also enable scanning a HashMap without relying on the limited functionality of the iterators returned by iter/iter_mut.

This addition would probably also require the raw_capacity function to be made public so users could know how many possible indices there were.

matklad · Jul 8, 2018

Is this API available in some extern crate by any chance? :-)

I'd like to see if I can implement a string interner on top of it as

struct Interner {
    /// Concatenation of all interned strings
    data: String
    /// A `HashSet<&'data str>` of interned strings, 
    /// but without those annoying ( :) ) lifetimes.
    mapping: HashSet<(u32, u32)>
}

Gankro · Jul 10, 2018

@fintelia no that would expose too many implementation details that we're not willing to commit to.

fintelia · Jul 10, 2018

@Gankro could you clarify which implementation details you'd be concerned about committing to? The very first line of the HashMap documentation already states that it is:

A hash map implemented with linear probing and Robin Hood bucket stealing.

Unless I'm missing something, it seems like that should be sufficient to guarantee that there is at least some way to map from indices to raw entries.

sfackler · Jul 11, 2018

That sentence is documenting the current implementation, not guaranteeing that will be the implementation for all time.

Mark-Simulacrum · Jul 25, 2018

@Amanieu Have you had a chance to look at taking this PR and fixing the remaining problems with it?

Amanieu · Jul 25, 2018

@Mark-Simulacrum Actually after talking with @Gankro I won't be working on this. Someone else can take over if they want.

jonhoo · Jul 26, 2018

@fintelia maybe this is something you'd be interested in picking up? Would save you from all the upkeep of rahashmap :p

fintelia · Jul 26, 2018

fintelia reviewed Jul 26, 2018

View changes

fintelia · Jul 26, 2018

src/libstd/collections/hash/map.rs

+    where M: DerefMut<Target = RawTable<K, V>>,
+          F: FnMut(&mut K) -> bool
+{
+    // This is the only function where capacity can be zero. To avoid


This function and the original search_hashed functions are the only two where capacity can be zero. Both comments should be updated...

fintelia · Jul 26, 2018

I'm willing to take over this PR. What is still remaining to do?

@jonhoo sadly, the PR in the current form cannot support rahashmap's central feature: there is no way to get a random element from the map. Specifically, a raw_entry can't be constructed without knowing its exact hash in advance, though if RawVacantEntry were to have a next() function that might be enough...

fintelia · Aug 30, 2018

@alexcrichton @Gankro What is required for me to take over this PR?

SimonSapin · Aug 30, 2018

@fintelia The easiest is probably to fetch this PR’s branch, rebase/rework it as needed, open a new PR, and link the new PR from here. Thanks for volunteering!

Gankro · Aug 31, 2018

I think my comments/checklist hold accurate

WIP: add raw_entry API to HashMap

Loading status checks…

ba33f6a

rust-highfive assigned withoutboats May 17, 2018

rust-highfive added the S-waiting-on-review label May 17, 2018

Gankro requested a review from alexcrichton May 17, 2018

rust-highfive assigned alexcrichton and unassigned withoutboats May 21, 2018

progress on raw_entry

Loading status checks…

0ef4970

fixup Debug bounds

Loading status checks…

fd4593f

disambiguate hashes

Loading status checks…

8ec7bdd

TimNN added S-waiting-on-author and removed S-waiting-on-review labels May 29, 2018

TimNN closed this Jun 12, 2018

kennytm added S-inactive-closed and removed S-waiting-on-author labels Jul 10, 2018

fintelia referenced this pull request Sep 7, 2018
Merged
Add raw_entry API to HashMap #54043

bors added a commit that referenced this pull request Nov 1, 2018

Auto merge of #54043 - fintelia:raw_entry, r=alexcrichton
Add raw_entry API to HashMap This is a continuation of #50821.

Loading status checks…

14331aa

bors added a commit that referenced this pull request Nov 2, 2018

Auto merge of #54043 - fintelia:raw_entry, r=alexcrichton
Add raw_entry API to HashMap This is a continuation of #50821.

Loading status checks…

e800988

rust-lang/rust

Join GitHub today

WIP: add raw_entry API to HashMap #50821

Conversation

Gankro commented May 17, 2018 • edited

rust-highfive assigned withoutboats May 17, 2018

This comment has been minimized.

rust-highfive commented May 17, 2018

rust-highfive added the S-waiting-on-review label May 17, 2018

This comment has been minimized.

withoutboats commented May 17, 2018

This comment has been minimized.

Gankro commented May 17, 2018

Gankro requested a review from alexcrichton May 17, 2018

This comment has been minimized.

Gankro commented May 17, 2018

This comment has been minimized.

Gankro commented May 17, 2018

This comment has been minimized.

pietroalbini commented May 21, 2018

rust-highfive assigned alexcrichton and unassigned withoutboats May 21, 2018

alexcrichton reviewed May 25, 2018 View changes

alexcrichton left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rust-highfive commented May 25, 2018

This comment has been minimized.

rust-highfive commented May 25, 2018

This comment has been minimized.

rust-highfive commented May 26, 2018

TimNN added S-waiting-on-author and removed S-waiting-on-review labels May 29, 2018

This comment has been minimized.

shepmaster commented Jun 2, 2018

This comment has been minimized.

TimNN commented Jun 12, 2018

TimNN closed this Jun 12, 2018

This comment has been minimized.

Gankro commented Jun 14, 2018

This comment has been minimized.

Amanieu commented Jul 3, 2018

fintelia reviewed Jul 3, 2018 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Gankro commented Jul 5, 2018

This comment has been minimized.

Gankro commented Jul 5, 2018

fintelia reviewed Jul 5, 2018 View changes

This comment has been minimized.

This comment has been minimized.

fintelia commented Jul 5, 2018

This comment has been minimized.

matklad commented Jul 8, 2018

This comment has been minimized.

Gankro commented Jul 10, 2018

kennytm added S-inactive-closed and removed S-waiting-on-author labels Jul 10, 2018

This comment has been minimized.

fintelia commented Jul 10, 2018

This comment has been minimized.

sfackler commented Jul 11, 2018

This comment has been minimized.

Mark-Simulacrum commented Jul 25, 2018

This comment has been minimized.

Amanieu commented Jul 25, 2018

This comment has been minimized.

jonhoo commented Jul 26, 2018 • edited

fintelia reviewed Jul 26, 2018 View changes

This comment has been minimized.

This comment has been minimized.

fintelia commented Jul 26, 2018

This comment has been minimized.

fintelia commented Aug 30, 2018

Gankro commented May 17, 2018 •

edited

alexcrichton reviewed May 25, 2018

View changes

fintelia reviewed Jul 3, 2018

View changes

fintelia reviewed Jul 5, 2018

View changes

jonhoo commented Jul 26, 2018 •

edited

fintelia reviewed Jul 26, 2018

View changes