Add XxHash3 to System.IO.Hashing #76641

stephentoub · 2022-10-04T21:19:48Z

[Params(2, 4, 8, 16, 32, 64, 128, 1024 * 1024)]
public int Count { get; set; }

private byte[] _output = new byte[100];
private byte[] _input;

[GlobalSetup]
public void Setup() => _input = RandomNumberGenerator.GetBytes(Count);

[Benchmark(Baseline = true)] public int XXH32_Hash() => XxHash32.Hash(_input, _output);
[Benchmark] public int XXH64_Hash() => XxHash64.Hash(_input, _output);
[Benchmark] public int XXH3_Hash() => XxHash3.Hash(_input, _output);
[Benchmark] public int Crc32_Hash() => Crc32.Hash(_input, _output);
[Benchmark] public int Crc64_Hash() => Crc64.Hash(_input, _output);

Method	Count	Mean	Ratio
Crc32_Hash	2	5.262 ns	0.75
Crc64_Hash	2	5.800 ns	0.83
XXH32_Hash	2	7.018 ns	1.00
XXH64_Hash	2	8.436 ns	1.20
XXH3_Hash	2	4.505 ns	0.64

Crc32_Hash	4	9.045 ns	1.35
Crc64_Hash	4	8.307 ns	1.24
XXH32_Hash	4	6.680 ns	1.00
XXH64_Hash	4	7.629 ns	1.14
XXH3_Hash	4	4.441 ns	0.66

Crc32_Hash	8	12.919 ns	1.70
Crc64_Hash	8	13.166 ns	1.74
XXH32_Hash	8	7.596 ns	1.00
XXH64_Hash	8	8.807 ns	1.16
XXH3_Hash	8	4.429 ns	0.59

Crc32_Hash	16	28.178 ns	2.73
Crc64_Hash	16	24.868 ns	2.41
XXH32_Hash	16	10.321 ns	1.00
XXH64_Hash	16	9.738 ns	0.95
XXH3_Hash	16	4.439 ns	0.43

Crc32_Hash	32	63.266 ns	4.52
Crc64_Hash	32	54.408 ns	3.88
XXH32_Hash	32	13.891 ns	1.00
XXH64_Hash	32	15.562 ns	1.12
XXH3_Hash	32	8.930 ns	0.64

Crc32_Hash	64	136.389 ns	6.94
Crc64_Hash	64	117.304 ns	5.91
XXH32_Hash	64	19.861 ns	1.00
XXH64_Hash	64	18.979 ns	0.95
XXH3_Hash	64	11.738 ns	0.59

Crc32_Hash	128	270.159 ns	8.05
Crc64_Hash	128	239.804 ns	7.14
XXH32_Hash	128	33.325 ns	1.00
XXH64_Hash	128	30.156 ns	0.89
XXH3_Hash	128	22.173 ns	0.66

Crc32_Hash	1048576	2,353,496.267 ns	10.30
Crc64_Hash	1048576	2,020,543.281 ns	8.82
XXH32_Hash	1048576	228,092.225 ns	1.00
XXH64_Hash	1048576	127,780.163 ns	0.56
XXH3_Hash	1048576	63,425.388 ns	0.28

ghost · 2022-10-04T21:20:06Z

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #75948

[Params(2, 4, 8, 16, 32, 64, 128, 1024 * 1024)]
public int Count { get; set; }

private byte[] _output = new byte[100];
private byte[] _input;

[GlobalSetup]
public void Setup() => _input = RandomNumberGenerator.GetBytes(Count);

[Benchmark(Baseline = true)] public int XXH32_Hash() => XxHash32.Hash(_input, _output);
[Benchmark] public int XXH64_Hash() => XxHash64.Hash(_input, _output);
[Benchmark] public int XXH3_Hash() => XxHash3.Hash(_input, _output);
[Benchmark] public int Crc32_Hash() => Crc32.Hash(_input, _output);
[Benchmark] public int Crc64_Hash() => Crc64.Hash(_input, _output);

Method	Count	Mean	Ratio
Crc32_Hash	2	5.262 ns	0.75
Crc64_Hash	2	5.800 ns	0.83
XXH32_Hash	2	7.018 ns	1.00
XXH64_Hash	2	8.436 ns	1.20
XXH3_Hash	2	4.505 ns	0.64

Crc32_Hash	4	9.045 ns	1.35
Crc64_Hash	4	8.307 ns	1.24
XXH32_Hash	4	6.680 ns	1.00
XXH64_Hash	4	7.629 ns	1.14
XXH3_Hash	4	4.441 ns	0.66

Crc32_Hash	8	12.919 ns	1.70
Crc64_Hash	8	13.166 ns	1.74
XXH32_Hash	8	7.596 ns	1.00
XXH64_Hash	8	8.807 ns	1.16
XXH3_Hash	8	4.429 ns	0.59

Crc32_Hash	16	28.178 ns	2.73
Crc64_Hash	16	24.868 ns	2.41
XXH32_Hash	16	10.321 ns	1.00
XXH64_Hash	16	9.738 ns	0.95
XXH3_Hash	16	4.439 ns	0.43

Crc32_Hash	32	63.266 ns	4.52
Crc64_Hash	32	54.408 ns	3.88
XXH32_Hash	32	13.891 ns	1.00
XXH64_Hash	32	15.562 ns	1.12
XXH3_Hash	32	8.930 ns	0.64

Crc32_Hash	64	136.389 ns	6.94
Crc64_Hash	64	117.304 ns	5.91
XXH32_Hash	64	19.861 ns	1.00
XXH64_Hash	64	18.979 ns	0.95
XXH3_Hash	64	11.738 ns	0.59

Crc32_Hash	128	270.159 ns	8.05
Crc64_Hash	128	239.804 ns	7.14
XXH32_Hash	128	33.325 ns	1.00
XXH64_Hash	128	30.156 ns	0.89
XXH3_Hash	128	22.173 ns	0.66

Crc32_Hash	1048576	2,353,496.267 ns	10.30
Crc64_Hash	1048576	2,020,543.281 ns	8.82
XXH32_Hash	1048576	228,092.225 ns	1.00
XXH64_Hash	1048576	127,780.163 ns	0.56
XXH3_Hash	1048576	63,425.388 ns	0.28

Author:	stephentoub
Assignees:	stephentoub
Labels:	`area-System.IO`
Milestone:	-

stephentoub · 2022-10-04T21:22:54Z

src/libraries/System.IO.Hashing/src/System/IO/Hashing/XxHash3.cs

+                    Vector128<uint> sourceVec = Vector128.LoadUnsafe(ref sourceRef);
+                    Vector128<uint> sourceKey = sourceVec ^ Vector128.LoadUnsafe(ref keyRef);
+
+                    // TODO: Figure out how to unwind this shuffle and just use Vector128.Multiply


@tannergooding, per our conversation yesterday, I was able to get rid of the other direct usage of Sse2.Multiply with a shuffle done before it, but I couldn't quite wrap my mind around getting rid of this one. If it's obvious to you how, please let me know :)

src/libraries/System.IO.Hashing/src/System/IO/Hashing/XxHash3.cs

EgorBo · 2022-10-04T22:23:31Z

src/libraries/System.IO.Hashing/src/System/IO/Hashing/XxHash3.cs

+                    Vector256<ulong> sum = accVec + sourceSwap.AsUInt64();
+                    Vector256<ulong> product = Avx2.IsSupported ?
+                        Avx2.Multiply(sourceKey, sourceKeyLow) :
+                        (sourceKey & Vector256.Create(~0u, 0u, ~0u, 0u, ~0u, 0u, ~0u, 0u)).AsUInt64() * (sourceKeyLow & Vector256.Create(~0u, 0u, ~0u, 0u, ~0u, 0u, ~0u, 0u)).AsUInt64();


@tannergooding is operator * for Vector128<ulong> supported on ARM64? judging by the codegen it is not (fallbacks to software impl)

(my comment points to Vector256 but I meant Vector128)

vcsjones · 2022-10-12T13:39:38Z

Just to point something out, XxHash3Tests.cs is basically unusable in Visual Studio Code. It causes it to lockup after a few seconds of scrolling. I don't believe this is any particular extension causing it, as I can reproduce it with code --disable-extensions.

I realize this is probably an upstream issue in Code which I will file an issue for, but I am unsure if or how that will be addressable.

stephentoub · 2022-10-12T13:42:50Z

Thanks. I'm not sure what to do about that, other than a) reduce the number of tests (I copied these from an internal set of tests from another XXH3 implementation) or b) separate the tests out into a data file that the test then reads from, which feels a bit arbitrary.

vcsjones · 2022-10-12T15:07:37Z

Filed an issue at microsoft/vscode#163434. Largely the problem seems to be lines 3870-3902. When I get rid of the extremely long lines, things go back to being snappy. When I disable syntax highlighting it works too, so it appears to be part of the C# language syntax highlighting.

I guess if there were anything to do in this pull request it would be to manually wrap those very long base64 strings from "AAAAAAAA" to "AAAA" + <newline> "AAAAA" so that the lines are < 10,000 characters long. That seems to work for me.

stephentoub · 2022-10-12T15:30:03Z

Thanks. Done. Better?

vcsjones · 2022-10-12T15:30:51Z

Thanks. Done. Better?

Much! Thank you!

ghost · 2022-10-20T13:28:44Z

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #75948

[Params(2, 4, 8, 16, 32, 64, 128, 1024 * 1024)]
public int Count { get; set; }

private byte[] _output = new byte[100];
private byte[] _input;

[GlobalSetup]
public void Setup() => _input = RandomNumberGenerator.GetBytes(Count);

[Benchmark(Baseline = true)] public int XXH32_Hash() => XxHash32.Hash(_input, _output);
[Benchmark] public int XXH64_Hash() => XxHash64.Hash(_input, _output);
[Benchmark] public int XXH3_Hash() => XxHash3.Hash(_input, _output);
[Benchmark] public int Crc32_Hash() => Crc32.Hash(_input, _output);
[Benchmark] public int Crc64_Hash() => Crc64.Hash(_input, _output);

Method	Count	Mean	Ratio
Crc32_Hash	2	5.262 ns	0.75
Crc64_Hash	2	5.800 ns	0.83
XXH32_Hash	2	7.018 ns	1.00
XXH64_Hash	2	8.436 ns	1.20
XXH3_Hash	2	4.505 ns	0.64

Crc32_Hash	4	9.045 ns	1.35
Crc64_Hash	4	8.307 ns	1.24
XXH32_Hash	4	6.680 ns	1.00
XXH64_Hash	4	7.629 ns	1.14
XXH3_Hash	4	4.441 ns	0.66

Crc32_Hash	8	12.919 ns	1.70
Crc64_Hash	8	13.166 ns	1.74
XXH32_Hash	8	7.596 ns	1.00
XXH64_Hash	8	8.807 ns	1.16
XXH3_Hash	8	4.429 ns	0.59

Crc32_Hash	16	28.178 ns	2.73
Crc64_Hash	16	24.868 ns	2.41
XXH32_Hash	16	10.321 ns	1.00
XXH64_Hash	16	9.738 ns	0.95
XXH3_Hash	16	4.439 ns	0.43

Crc32_Hash	32	63.266 ns	4.52
Crc64_Hash	32	54.408 ns	3.88
XXH32_Hash	32	13.891 ns	1.00
XXH64_Hash	32	15.562 ns	1.12
XXH3_Hash	32	8.930 ns	0.64

Crc32_Hash	64	136.389 ns	6.94
Crc64_Hash	64	117.304 ns	5.91
XXH32_Hash	64	19.861 ns	1.00
XXH64_Hash	64	18.979 ns	0.95
XXH3_Hash	64	11.738 ns	0.59

Crc32_Hash	128	270.159 ns	8.05
Crc64_Hash	128	239.804 ns	7.14
XXH32_Hash	128	33.325 ns	1.00
XXH64_Hash	128	30.156 ns	0.89
XXH3_Hash	128	22.173 ns	0.66

Crc32_Hash	1048576	2,353,496.267 ns	10.30
Crc64_Hash	1048576	2,020,543.281 ns	8.82
XXH32_Hash	1048576	228,092.225 ns	1.00
XXH64_Hash	1048576	127,780.163 ns	0.56
XXH3_Hash	1048576	63,425.388 ns	0.28

Author:	stephentoub
Assignees:	stephentoub
Labels:	`area-System.Security`, `new-api-needs-documentation`
Milestone:	-

src/libraries/System.IO.Hashing/tests/System.IO.Hashing.Tests.csproj

stephentoub · 2022-10-25T00:23:32Z

@bartonjs, any other feedback?

bartonjs · 2022-10-25T00:33:37Z

@bartonjs, any other feedback?

Well, at a high level, I'd love to get rid of all of the pointers; though maybe the use of fixed in the state is what's forcing it all.

Otherwise it was a bit intense for my low-coffee-nated brain this morning and I didn't remember to look at it again today; I'll try to get to it tomorrow afternoon.

stephentoub · 2022-10-25T00:47:38Z

though maybe the use of fixed in the state is what's forcing it all.

That's part of it. The other part of it is that it's a whole lot faster with the pointers; the way I initially had it, just indexing into spans, the bounds check costs were measurable. They could be eliminated by the JIT if enough things were aggressively inlined so that it could see earlier length checks, but that then resulted in significant size bloat as well as hitting up against inlining limits.

I think the use of pointers here is reasonable. It's localized and it matches the reference implementation.

I'll try to get to it tomorrow afternoon.

Thanks!

bartonjs

I've made some progress, will have to finish tomorrow.

src/libraries/System.IO.Hashing/src/System/IO/Hashing/XxHash3.cs

stephentoub · 2022-10-27T01:44:23Z

Thanks for the helpful review, @bartonjs. I addressed all your feedback.

dotnet-issue-labeler bot added the area-System.IO label Oct 4, 2022

ghost assigned stephentoub Oct 4, 2022

stephentoub force-pushed the xxh3 branch from d453a23 to 844955e Compare October 4, 2022 21:21

stephentoub commented Oct 4, 2022

View reviewed changes

EgorBo reviewed Oct 4, 2022

View reviewed changes

src/libraries/System.IO.Hashing/src/System/IO/Hashing/XxHash3.cs Outdated Show resolved Hide resolved

EgorBo reviewed Oct 4, 2022

View reviewed changes

src/libraries/System.IO.Hashing/src/System/IO/Hashing/XxHash3.cs Outdated Show resolved Hide resolved

EgorBo reviewed Oct 4, 2022

View reviewed changes

stephentoub closed this Oct 5, 2022

stephentoub reopened this Oct 5, 2022

EgorBo mentioned this pull request Oct 5, 2022

Suboptimal x64 codegen for signed Math.BigMul #75594

Open

stephentoub force-pushed the xxh3 branch from 844955e to 4dec4dd Compare October 9, 2022 02:09

stephentoub requested review from GrabYourPitchforks and bartonjs October 10, 2022 17:05

This was referenced Oct 11, 2022

Tracking Nuget 429s dotnet/arcade#10885

Closed

Tracking issue for CI build timeouts #76454

Closed

stephentoub added the new-api-needs-documentation label Oct 11, 2022

stephentoub force-pushed the xxh3 branch from 20d229c to 2f8c6d9 Compare October 12, 2022 15:28

adamsitnik added area-System.Security and removed area-System.IO labels Oct 20, 2022

bartonjs reviewed Oct 24, 2022

View reviewed changes

src/libraries/System.IO.Hashing/tests/System.IO.Hashing.Tests.csproj Show resolved Hide resolved

stephentoub commented Oct 24, 2022

View reviewed changes

src/libraries/System.IO.Hashing/tests/System.IO.Hashing.Tests.csproj Show resolved Hide resolved

bartonjs reviewed Oct 26, 2022

View reviewed changes

src/libraries/System.IO.Hashing/src/System/IO/Hashing/XxHash3.cs Outdated Show resolved Hide resolved

bartonjs reviewed Oct 26, 2022

View reviewed changes

stephentoub and others added 4 commits October 26, 2022 17:47

Add XxHash3

ead1c9c

Add more XxHash3 test cases

73b76e4

Cull back volume of test cases, and line wrap to help editor performance

836456a

Address PR feedback

c3056f7

stephentoub force-pushed the xxh3 branch from ad6161e to c3056f7 Compare October 27, 2022 01:43

bartonjs approved these changes Oct 27, 2022

View reviewed changes

stephentoub merged commit 1adda68 into dotnet:main Oct 27, 2022

stephentoub deleted the xxh3 branch October 27, 2022 20:57

EgorBo mentioned this pull request Nov 1, 2022

Optimize XxHash3 #77756

Merged

vcsjones mentioned this pull request Nov 5, 2022

Add XxHash128 #77944

Merged

jeffhandley added area-System.IO.Hashing and removed area-System.Security labels Nov 22, 2022

dotnet locked as resolved and limited conversation to collaborators Dec 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add XxHash3 to System.IO.Hashing #76641

Add XxHash3 to System.IO.Hashing #76641

stephentoub commented Oct 4, 2022

ghost commented Oct 4, 2022

stephentoub Oct 4, 2022

EgorBo Oct 4, 2022 •

edited

vcsjones commented Oct 12, 2022

stephentoub commented Oct 12, 2022 •

edited

vcsjones commented Oct 12, 2022

stephentoub commented Oct 12, 2022

vcsjones commented Oct 12, 2022

ghost commented Oct 20, 2022

stephentoub commented Oct 25, 2022

bartonjs commented Oct 25, 2022

stephentoub commented Oct 25, 2022

bartonjs left a comment

stephentoub commented Oct 27, 2022

Add XxHash3 to System.IO.Hashing #76641

Add XxHash3 to System.IO.Hashing #76641

Conversation

stephentoub commented Oct 4, 2022

ghost commented Oct 4, 2022

stephentoub Oct 4, 2022

Choose a reason for hiding this comment

EgorBo Oct 4, 2022 • edited

Choose a reason for hiding this comment

vcsjones commented Oct 12, 2022

stephentoub commented Oct 12, 2022 • edited

vcsjones commented Oct 12, 2022

stephentoub commented Oct 12, 2022

vcsjones commented Oct 12, 2022

ghost commented Oct 20, 2022

stephentoub commented Oct 25, 2022

bartonjs commented Oct 25, 2022

stephentoub commented Oct 25, 2022

bartonjs left a comment

Choose a reason for hiding this comment

stephentoub commented Oct 27, 2022

EgorBo Oct 4, 2022 •

edited

stephentoub commented Oct 12, 2022 •

edited