Change Siphash to use one of the faster variants of the algorithm (Siphash13, Highwayhash)

[funny-falcon]

Jean-Philippe Aumasson answered on "can Siphash13 used instead of Siphash24 for hash function of hash tables":

we proposed SipHash-2-4 as a (strong) PRF/MAC, and so far no attack whatsoever has been found,
although many competent people tried to break it. However, fewer rounds may be sufficient and I would
be very surprised if SipHash-1-3 introduced weaknesses for hash tables.

So that, there is no reason to burn more cpu power on redundant rounds.

[bstrie]

Sounds like it would only take a two-line change to test the perf impact of this change. @gankro, did you have some hashtable benchmark scripts lying around?

[arthurprs]

That'd be interesting. Sub.

[Gankra]

You can fork http://cglab.ca/~abeinges/blah/hash-rs/ if you want to do some ok-ish comparisons.

@bluss is my goto expert on siphash.

[Gankra]

Note that direct consumers of SipHash may break if they decided to manually seed it and store the results. So this isn't a totally free change.

[bluss]

Aumasson is the cryptographer, maybe we can ask him what he thinks about Rust going for SipHash-1-3 specifically? I'm curious what people have learned about SipHash in general now that it's been out for several years.

If we weaken it so much that it doesn't prevent hash flooding anymore, then we lose the reason to use SipHash and could use a faster hash function.

I know I tried long ago and noted that SipHash-1-2 didn't even pass the Smhasher testsuite (which is probably bad). (SipHash-N-M is for N hash rounds per 8 bytes of input and M fixed rounds as finalization).

[funny-falcon]

Siphash-1-3 passes Smhasher. It is smallest configuration, which has good avalanche.

Siphash is protected from "seed independent collisions" with any configuration with good avalanche.

Siphash24 is recomended for MAC cause when you do MAC you put result in public, so there are other vectors of attack: plain text, chosen plain text, differential, etc. As cryptographers, Siphash authors simply recommend more rounds than minimal required to protect against future investigations.

Within hash-table result of hash-function is not published, so there is no way to recover seed. Then protection from "seed independent collisions" is just enough.

Note that direct consumers of SipHash may break if they decided to manually seed it and store the results. So this isn't a totally free change.

Agree. There should be separate SipHash13 for internal/hash-table usage, and SipHash (24) remains for direct/external usage.

[funny-falcon]

Aumasson is the cryptographer, maybe we can ask him what he thinks about Rust going for SipHash-1-3 specifically?

I asked him year ago about using SipHash13 for hash tables. Answer is in issue text.

I think, we can ask him again.

[tarcieri]

If @veorq feels its fine then it should be fine. The worst case is there's a hashDoS against Siphash13, in which case you can upgrade to a stronger hash function.

[bstrie]

@gankro, how likely do you think this is to break code in the wild? I was under the impression that we considered the precise underlying default hash function to be an implementation detail (with the caveat that you can count on it to be cryptographically secure).

[tarcieri]

@bstrie from a security perspective, I'd say you definitely should not couple to the hash function in any way. Java made that mistake (the hashing algorithm was included in the Java Language Specification) and that made it so they were not able to adequately respond to hashDoS.

[bstrie]

@tarcieri That's actually tied in with what I'm implying here. Supposing that we did switch to 1-3, and that an attack on 1-3 was later discovered, we'd want to reserve the right to bump it back up to 2-4 (or, you know, whatever the state of the art is by then) without risking massive breakage in the ecosystem. If changing the default hash function should prove to be disruptive at the current point in time then that's a separate problem that we need to consider, hence why I'm curious as to the true magnitude of such a change.

Furthermore, I get the impression that the breakage that Gankro's envisioning wouldn't be detectable by Crater.

[bluss]

Using 1-3 doesn't fix the fundamental slowness of our hashing, but it will instead speed up the long data case. The short data case remains a problem, due to (I think):

• Lack of one-shot hashing, means the code visits branches to read and or update the partially written tail
• Strings and slices write both data and length / delimiter, which is slow since it enters SipHash::write twice, hitting the code in the first point.

Of course we should still go for it, if possible, but I don't think it's a substitute for allowing faster alternatives to be used (application may choose to sacrifice hash flooding defence, deciding it's not useful for them).

[tarcieri]

@bstrie it seems when Java was last affected by hashDoS they ultimately ended up making a breaking change to the language specification:

http://mail.openjdk.java.net/pipermail/core-libs-dev/2012-May/010238.html

[ranma42]

Did anybody experiment adding padding to 8 bytes in SipHash?
In particular, if we padded whatever is input to 8 bytes, it would be possible to remove the ntail handling code here

rust/src/libcore/hash/sip.rs

Lines 151 to 167 in e3cd872

	if self.ntail != 0 {
	needed = 8 - self.ntail;
	if length < needed {
	self.tail |= u8to64_le!(msg, 0, length) << 8 * self.ntail;
	self.ntail += length;
	return
	}
	let m = self.tail | u8to64_le!(msg, 0, needed) << 8 * self.ntail;
	self.v3 ^= m;
	compress!(self.v0, self.v1, self.v2, self.v3);
	compress!(self.v0, self.v1, self.v2, self.v3);
	self.v0 ^= m;
	self.ntail = 0;
	}

and

rust/src/libcore/hash/sip.rs

Lines 185 to 186 in e3cd872

	self.tail = u8to64_le!(msg, i, left);
	self.ntail = left;

[veorq]

SipHash designer here, haven't changed my opinion about SipHash-1-3 :-)

A warning, though: SipHash-1-3 leaves 4 rounds between the last attacker-controlled input and the output. There's a "distinguisher" on 4 rounds in https://eprint.iacr.org/2014/722.pdf, or in simplest terms a statistical bias that shows up given a specific difference pattern in the input of the 4-round sequence. But you can't inject that pattern in SipHash-1-3 because you don't control all the state. And even if you could inject that pattern the bias wouldn't be exploitable anyway.

[arthurprs]

@bluss yeah, sooner of later that will need to be addressed as well.

[bstrie]

@veorq, thanks for your input!

@bluss, indeed I don't believe anybody here is suggesting this would obviate the desired ability to provide an alternative hashing algorithm.

[bstrie]

@ranma42, would you be so kind as to open an issue for that? It likely deserves its own discussion and shouldn't get tangled up with this one.

@tarcieri, do you know of any experience reports from the fallout of that change in the broader Java ecosystem?

[arthurprs]

I benchmarked it with @gankro machinery, it definitely improves the speed. See https://i.imgur.com/5dKecOW.jpg

As we got confirmation it's secure enough it looks like a worthy improvement.

[Gankra]

Yeah, looks good. Only worry is people "inappropriately" depending on the algorithm being stable.

[arthurprs]

Can't we include this variant as SipHasher13 and set it as the default hasher? I don't see how this can break anybodys code. Am I missing something?

[bluss]

I don't think we should change std::hash::SipHash, since it's stable and documented to be SipHash-2-4. We can keep the new hasher private for now.

[bstrie]

@bluss I can imagine support for an RFC to deprecate std::hash::SipHash and replace it with std::hash::SipHash24 and std::hash::SipHash13.

[bstrie]

@gankro, I'm leaning towards saying that this warrants an RFC, unless you think breakage will be essentially nanoscule. (Did we really document that our SipHash variant is specifically 2-4, and did we make it sound like that's guaranteed? If so, that's worth changing too.)

[bluss]

I think our HashMap is completely free to pick its hash function. Changing it should not be a problem.

[Gankra]

SipHash 2-4 is, as far as I can tell, "the" SipHash, so it's totally sound for it to be exposed as SipHash. Having WeakSipHash or whatevs should be fine.

[arthurprs]

@briansmith sounds like a good idea to me.

[ticki]

I've been working on trhasher, which can be used to benchmark such an implementation.

[funny-falcon]

@briansmith cause SipHash-1-3 is already secure enough for any usage in hash table.
More over, it is still not broken for any other usage, so at this moment of time, it is still cryptographically secure.
It is better to use "faster implementation of SipHash-1-3", than "faster implementation of SipHash-2-4".