Use std::unordered_set instead of set in blockfilter interface #14074

jimpo · 2018-08-26T19:04:08Z

Use std::unordered_set (hash set) instead of std::set (tree set) in blockfilter interface, as suggested by @ryanofsky in #12254. This may result in a very minor speedup, but I haven't measured.

This moves CSipHasher to it's own file crypto/siphash.h, so that it can be used in the libbitcoin_util library without including hash.{h,cpp}. I'm open to other suggestions on solving this issue if people would prefer to leave CSipHasher where it is.

DrahtBot · 2018-08-26T20:31:18Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#14387 (Faster Input Deduplication Algorithm by JeremyRubin)
#14224 (Document intentional and unintentional unsigned integer overflows (wraparounds) using annotations by practicalswift)
#14121 (Index for BIP 157 block filters by jimpo)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

laanwj · 2018-08-27T18:41:15Z

src/util.h

+public:
+    ByteVectorHash();
+    size_t operator()(const std::vector<unsigned char>& input) const;
+};


I don't think util.h is the best place for a generic utility data structure such as this, it's more for assorted operating system functions

@laanwj What would be the right place? A new file utiltypes.{h,cpp}? Or util/types.{h,cpp}?

or maybe support/bytevectorhash.h ?
unless there's a strong need to group this with other things

laanwj · 2018-08-27T18:41:49Z

src/crypto/siphash.h

@@ -0,0 +1,47 @@
+// Copyright (c) 2016-2018 The Bitcoin Core developers


ACK on moving siphash to a separate unit

laanwj · 2018-08-31T12:51:57Z

utACK f4608359e4e643c6e197df822e03226146e37a49
verified move-onlyness of extraction commit

src/support/bytevectorhash.cpp

practicalswift · 2018-09-10T06:41:59Z

src/crypto/siphash.cpp

+    v[2] = 0x6c7967656e657261ULL ^ k0;
+    v[3] = 0x7465646279746573ULL ^ k1;
+    count = 0;
+    tmp = 0;


Initialize count and tmp using default member initializers or in the member initializer list of the constructor?

This is a move-only commit. I'd rather not modify the contents of the code here.

Makes sense!

laanwj · 2018-09-10T20:11:48Z

@ryanofsky can you take a look here please? according to @jimpo this was your suggestion

promag

@jimpo can you improve PR description of the advantages of this change (beside linking @ryanofsky suggestion)?

Do you have numbers to support this change?

Left some nits.

promag · 2018-10-08T21:37:37Z

src/support/bytevectorhash.h

+#define BITCOIN_SUPPORT_BYTEVECTORHASH_H
+
+/**
+ * Implementation of Hash named requirement for a byte vector. This may be used the hash function in


This may be used as the hash.. ?

promag · 2018-10-08T21:41:28Z

src/support/bytevectorhash.h

+ */
+class ByteVectorHash
+{
+private:


Drop private.

I think it's more clear to leave it explicit even if the default for class is private.

promag · 2018-10-08T21:50:08Z

src/txmempool.h

@@ -15,6 +15,7 @@

 #include <amount.h>
 #include <coins.h>
+#include <crypto/siphash.h>


Can be removed?

It's used in SaltedTxidHasher

promag · 2018-10-08T21:56:32Z

src/util.cpp

@@ -6,6 +6,7 @@
 #include <util.h>

 #include <chainparamsbase.h>
+#include <crypto/siphash.h>


Can be removed?

promag · 2018-10-08T21:57:02Z

src/support/bytevectorhash.h

+#ifndef BITCOIN_SUPPORT_BYTEVECTORHASH_H
+#define BITCOIN_SUPPORT_BYTEVECTORHASH_H
+
+/**


Missing includes <stdint.h> and <vector>.

promag · 2018-10-08T21:58:28Z

src/support/bytevectorhash.h

+ * std::unordered_set or std::unordered_map over std::vector<unsigned char>. Internally, a random
+ * instance of SipHash-2-4 is used.
+ */
+class ByteVectorHash


nit, final.

ryanofsky

utACK f4608359e4e643c6e197df822e03226146e37a49. Sorry for missing this previously. I confirmed the first commit is move-only, and I think the changes in the second commit should make it easier to use hash maps more places in the future. I left a suggestion below, but also think this change looks good as it is.

ryanofsky · 2018-10-09T00:58:08Z

src/support/bytevectorhash.h

+
+/**
+ * Implementation of Hash named requirement for a byte vector. This may be used the hash function in
+ * std::unordered_set or std::unordered_map over std::vector<unsigned char>. Internally, a random


In commit "blockfilter: Use unordered_set instead of set in blockfilter." (f4608359e4e643c6e197df822e03226146e37a49)

The way this is written makes it seem like this whole class is tied to std::vector<unsigned char>, when actually only the call operator is. I guess my concern that if someone wants to reuse this hash function on a similar type like Span<char>, prevector, or std::string, the naming and comments here will lead them to copy/paste/rename this whole class instead of doing something simpler like overloading the call operator or adding a Span conversion.

I'd suggest:

Mentioning in this comment that operator() overloads for new types could be added here in the future.

Changing the name of the class from ByteVectorHash to something like RandomizedSipHash to avoid tying it to std::vector.

Maybe replacing std::vector with Span.

I updated the comment to say that it works for any types that internally store (or reference) a byte array, but decided not to change the class name.

etscrivner · 2018-10-10T22:21:23Z

utACK cfabca8749a132220e649d4563fb876afc21ceec

sipa · 2018-10-11T09:38:36Z

Just one not comment: I always imagined the things in 'support' as low level features that are independently usable, and find it a bit strange to put something there that depends on random and crypto/siphash. Perhaps others have a different understanding, as I don't think it was ever written down somewhere what it was for, but I would just put it in src/.

jimpo · 2018-10-11T19:37:04Z

@sipa I'd prefer to start creating more directory structure. How would you feel about a util/ directory instead and we can put it in there?

sipa · 2018-10-12T20:01:22Z

@jimpo Sounds great, but perhaps something to do independently? I'd prefer to not start a new convention, and then have it linger in a half finished state if there's unclarity about how to move forward.

ryanofsky

utACK cfabca8749a132220e649d4563fb876afc21ceec. Only changes are updating ByteVectorHash comment, adding final specifier, and tweaking includes.

2068f08 scripted-diff: Move util files to separate directory. (Jim Posen) Pull request description: As discussed [here](#14074 (comment)), this establishes a `util/` directory to introduce more organizational structure and have a clear place for new util files. It's really not scary to review, it's just one big scripted diff. Tree-SHA512: 39cf15480d7d35e987b6088d52a857a2d5b1802e36c6b815eb42718d80cd95e669757af9bcc7c04426cd8523662cb1050b8da1e2377d3730672820ed298b894b

This is a move-only commit with the exception of changes to includes.

jimpo · 2018-11-05T23:12:48Z

@sipa I moved bytevectorhash to util/ now.

laanwj · 2018-11-06T14:36:04Z

utACK fef5adc

@ryanofsky

…terface fef5adc blockfilter: Use unordered_set instead of set in blockfilter. (Jim Posen) 4fb789e Extract CSipHasher to it's own file in crypto/ directory. (Jim Posen) Pull request description: Use `std::unordered_set` (hash set) instead of `std::set` (tree set) in blockfilter interface, as suggested by @ryanofsky in #12254. This may result in a very minor speedup, but I haven't measured. This moves `CSipHasher` to it's own file `crypto/siphash.h`, so that it can be used in the libbitcoin_util library without including `hash.{h,cpp}`. I'm open to other suggestions on solving this issue if people would prefer to leave CSipHasher where it is. Tree-SHA512: 593d1abda771e45f2860d5334272980d20df0b81925a402bb9ee875e17595c2517c0d8ac9c579218b84bbf66e15b49418241c1fe9f9265719bcd2377b0cd0d88

…nterface

…lter interface

partial bitcoin#15638, bitcoin#21966, bitcoin#16889, merge bitcoin#14555, bitcoin#20499, bitcoin#14074, bitcoin#17073: util refactoring

This was referenced Aug 26, 2018

utils: Convert Windows args to utf-8 string #13883

Merged

Docs: Create default bitcoin.conf file on startup #13761

Closed

-masterdatadir for datadir bootstrapping #13746

Closed

bitcoin-cli: -stdinwalletpassphrase and non-echo stdin passwords #13716

Merged

fanquake added the Refactoring label Aug 26, 2018

laanwj reviewed Aug 27, 2018

View reviewed changes

jimpo force-pushed the blockfilter-unordered-set branch from aa37070 to f460835 Compare August 28, 2018 18:48

DrahtBot mentioned this pull request Aug 31, 2018

Index for BIP 157 block filters #14121

Merged

practicalswift reviewed Sep 10, 2018

View reviewed changes

src/support/bytevectorhash.cpp Outdated Show resolved Hide resolved

practicalswift reviewed Sep 10, 2018

View reviewed changes

DrahtBot mentioned this pull request Sep 15, 2018

Document intentional and unintentional unsigned integer overflows (wraparounds) using annotations #14224

Closed

DrahtBot mentioned this pull request Oct 4, 2018

Faster Input Deduplication Algorithm #14387

Closed

promag reviewed Oct 8, 2018

View reviewed changes

ryanofsky approved these changes Oct 9, 2018

View reviewed changes

jimpo force-pushed the blockfilter-unordered-set branch 2 times, most recently from 7406129 to cfabca8 Compare October 10, 2018 18:22

ryanofsky reviewed Oct 19, 2018

View reviewed changes

jimpo mentioned this pull request Oct 23, 2018

Move util files to directory #14555

Merged

jimpo added 2 commits November 5, 2018 09:25

Extract CSipHasher to it's own file in crypto/ directory.

4fb789e

This is a move-only commit with the exception of changes to includes.

blockfilter: Use unordered_set instead of set in blockfilter.

fef5adc

jimpo force-pushed the blockfilter-unordered-set branch from cfabca8 to fef5adc Compare November 5, 2018 17:31

laanwj merged commit fef5adc into bitcoin:master Nov 6, 2018

jimpo deleted the blockfilter-unordered-set branch November 6, 2018 16:59

kwvg added a commit to kwvg/dash that referenced this pull request Jun 15, 2021

bitcoin#14074: Use std::unordered_set instead of set in blockfilter i…

904da1f

…nterface

kwvg added a commit to kwvg/dash that referenced this pull request Jun 15, 2021

bitcoin#14074: Use std::unordered_set instead of set in blockfilter i…

7c564a4

…nterface

kwvg added a commit to kwvg/dash that referenced this pull request Jun 16, 2021

merge bitcoin#14074: Use std::unordered_set instead of set in blockfi…

a4ee942

…lter interface

kwvg mentioned this pull request Jun 16, 2021

partial bitcoin#15638, #21966, #16889, merge #14555, #20499, #14074, #17073: util refactoring dashpay/dash#4197

Merged

kwvg added a commit to kwvg/dash that referenced this pull request Jun 25, 2021

merge bitcoin#14074: Use std::unordered_set instead of set in blockfi…

abf1c91

…lter interface

kwvg added a commit to kwvg/dash that referenced this pull request Jun 25, 2021

merge bitcoin#14074: Use std::unordered_set instead of set in blockfi…

f90ceb3

…lter interface

kwvg added a commit to kwvg/dash that referenced this pull request Jun 26, 2021

merge bitcoin#14074: Use std::unordered_set instead of set in blockfi…

fa92d7e

…lter interface

kwvg added a commit to kwvg/dash that referenced this pull request Jun 26, 2021

merge bitcoin#14074: Use std::unordered_set instead of set in blockfi…

67bb702

…lter interface

kwvg added a commit to kwvg/dash that referenced this pull request Jun 27, 2021

merge bitcoin#14074: Use std::unordered_set instead of set in blockfi…

9a5a285

…lter interface

kwvg added a commit to kwvg/dash that referenced this pull request Jun 27, 2021

merge bitcoin#14074: Use std::unordered_set instead of set in blockfi…

6803196

…lter interface

UdjinM6 added a commit to dashpay/dash that referenced this pull request Jun 27, 2021

Merge pull request #4197 from kittywhiskers/utilRefactor

7d664c7

partial bitcoin#15638, bitcoin#21966, bitcoin#16889, merge bitcoin#14555, bitcoin#20499, bitcoin#14074, bitcoin#17073: util refactoring

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use std::unordered_set instead of set in blockfilter interface #14074

Use std::unordered_set instead of set in blockfilter interface #14074

jimpo commented Aug 26, 2018 •

edited

Loading

DrahtBot commented Aug 26, 2018 •

edited

Loading

laanwj Aug 27, 2018

jimpo Aug 27, 2018

laanwj Aug 28, 2018

laanwj Aug 27, 2018

laanwj commented Aug 31, 2018

practicalswift Sep 10, 2018

jimpo Sep 10, 2018

practicalswift Sep 10, 2018

laanwj commented Sep 10, 2018

promag left a comment

promag Oct 8, 2018

promag Oct 8, 2018

jimpo Oct 9, 2018

promag Oct 8, 2018

jimpo Oct 10, 2018

promag Oct 8, 2018

promag Oct 8, 2018

promag Oct 8, 2018

ryanofsky left a comment

ryanofsky Oct 9, 2018

jimpo Oct 10, 2018

etscrivner commented Oct 10, 2018

sipa commented Oct 11, 2018

jimpo commented Oct 11, 2018

sipa commented Oct 12, 2018 •

edited

Loading

ryanofsky left a comment

jimpo commented Nov 5, 2018

laanwj commented Nov 6, 2018

		@@ -0,0 +1,47 @@
		// Copyright (c) 2016-2018 The Bitcoin Core developers

Use std::unordered_set instead of set in blockfilter interface #14074

Use std::unordered_set instead of set in blockfilter interface #14074

Conversation

jimpo commented Aug 26, 2018 • edited Loading

DrahtBot commented Aug 26, 2018 • edited Loading

Conflicts

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laanwj commented Aug 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laanwj commented Sep 10, 2018

promag left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanofsky left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etscrivner commented Oct 10, 2018

sipa commented Oct 11, 2018

jimpo commented Oct 11, 2018

sipa commented Oct 12, 2018 • edited Loading

ryanofsky left a comment

Choose a reason for hiding this comment

jimpo commented Nov 5, 2018

laanwj commented Nov 6, 2018

jimpo commented Aug 26, 2018 •

edited

Loading

DrahtBot commented Aug 26, 2018 •

edited

Loading

sipa commented Oct 12, 2018 •

edited

Loading