hash: add Hasher, Seed for hashing user data

[alandonovan]

[See #28322 (comment) below for accepted proposal.]

Some applications need an efficient non-cryptographic hash function for strings. The Go runtime contains such a function, hashString, that is optimized to use AESENC instructions on amd64, but it is not exported. The implementation is non-trivial, so clients are loathe to copy it and will tend to use the //linkname hack instead.

In the same way that math/bits exports Go functions that use the best available implementation of a given bit-twiddling function, it would be useful for a package such as hash to export a decent string hash function. The implementation should include a per-process random salt to prevent users from assuming that hashes are stable across process restarts, which has been a problem for hash libraries in other languages.

package hash

// String computes a non-cryptographic hash of the specified string.
// The hash function is not specified, but callers may assume that,
// within a single process, x==y implies hash.String(x)==hash.String(y).
func String(string) uinptr

// ditto, bytes
func Bytes([]byte) uintptr

If we can agree where this belongs, I'll send the CL.

[dsnet]

Related to #21195.

[dgryski]

It seems to me this could easily be an external package. There are already many good string hashing functions available with slightly different tradeoffs: strength, speed for short strings, speed for long strings, etc. (I have a few benchmarks here: https://github.com/dgryski/trifles/blob/master/hashbench/bench_test.go#L10 ) A "one-size-fits-all" solution is nice, but I think this ends up being a sufficiently rare it shouldn't be in the standard library. That said, packaging a copy of the runtime's string hash function into /x/ somewhere might be an idea.

[rsc]

This is not the right API. All hash functions must take an initial seed so that they can be chained and produce a different hash function for different seeds. The current proposed API, with no input seed, can only be used to build bad hash tables. (To avoid engineered collisions aka hash-flooding it is important that when you grow a hash table you also completely change the hash function, which you can do by having a different initial seed for the hash inputs.) On top of that we'd also want to vary the hash calculation, even for a given input seed value, for each process, to avoid people writing code dependent on the exact hash implementation.

More API design is probably needed here.

It might work to copy this to an external package and simply have two of the code. On the other hand this functionality is very much exporting what's in the runtime (unexported), so that would weigh toward being in the standard library. The right answer may depend on how much API we are talking about.

[alandonovan]

Well, adding a seed parameter is trivial, and corresponds to the underlying implementation:

func String(s string, seed uintptr) uinptr

[andybons]

Since uintptr is architecture-dependent, this seems like it could lead to an awkward API. Would it better to pass a uint64 and on 32-bit architectures we take the perf hit?

Per discussion with @golang/proposal-review

[rsc]

Does anyone have time to work out a concrete API here? Or should we put this on hold?

[alandonovan]

Does the API need to be more complex that this?

package hash

// String computes a non-cryptographic hash of s.
// The hash function is not specified but callers may assume that,
// within a single process, x==y implies hash.String(x, seed)==hash.String(y, seed)
// for any seed value.
func String(s string, seed uintptr) uinptr

// ditto, bytes
func Bytes([]byte, seed uintptr) uintptr

[dsnet]

Is the compiler intelligent enough today to avoid allocating when doing:

var b []byte = ...
h := hash.String(string(b), seed)

such that we can avoid the bytes version?

[dgryski]

Also probably want to make it clear that different processes even with the same seed may produce different hashes.

[rsc]

Also should probably be uint64; uint32 is likely not good enough.

It's unclear to me if the API should be limited to a single function like this or if we want to define a Hasher object that you can feed strings and other basic types into.

[alandonovan]

A stateful hasher object that satisfies hash.Hash64 is a good idea: it makes the state parameter invisible in the common case, it presents an API consistent with the other hash/... packages, and it allows you to write non-strings to it using Fprintf.