Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
crypto/sha256: provide a way to efficiently hash multiple byte slices #21948
I'm trying to implement an optimized version of HMAC-SHA-256 digest computation that avoids allocation.
The function signature I provide is the following:
The MAC is computed over the list of data slices given as argument.
Unfortunately, I'm not able to use the hardware implementation of SHA, so I'm stuck.
I could use the
With a small backward compatible change to this function signature, it would be possible to compute an SHA over multiple slices without needing an allocation with sha256.New.
hmac requires to compute a hash over at least two data blocks.
If there is another solution to use the hardware or assembly optimized sha256 block function, I'm a buyer.
PS: as explained in the end of the readme of the github repository, my goal is to use this for secure cookie which has to be verified for nearly every request. Avoiding these basically useless allocations, could reduce the pressure on the GC.
Unfortunately we can not make that change, as it would violate the Go 1 compatibility guarantee (while existing calls would still compile, other uses of the function might not).
Unfortunately, I can't adapt the function Sum256 because it uses private functions and struct. I would have done it if it was so simple.
Could you explain me what it would break by changing the data argument to a vararg ? I honnestly don't know.
Edit: oh I see. By changing the function signature, the type of a function variable would change.
changed the title from
Request: change Sum256(data byte) to Sum256(data ...byte) in crypto/sha256 package
crypto/sha256: provide a way to efficiently hash multiple byte slices
Sep 20, 2017
Given this snippet:
var b1, b2, b3 byte var h [sha256.Size]byte d := sha256.New() d.Write(b1) d.Write(b2) d.Write(b3) d.Sum(h[:0])
@bsadfitz #21070 is about Sum but it address a different issue which I agree with. AppendSum would make clear that it uses the append pattern. It took me some time to figure it out. For #19361, I don't see the link with this issue.
A compiler optimization would allow to fix the problem for all hashes in one hit. But I have no idea of how much effort it would require. After thinking about it, I agree that adding functions would a false good idea.
PS: For the secure cookie, I think I can get around the problem by concatenating slices to hash. This should be possible by arranging the data differently with a minimum of data copy. So for my point of view, the possibility to hash multiple slices by avoiding a heap allocation is now "good to have" instead of "must have". Beside, hashing one block is faster than hashing multiple block fragment. So I hope the concatenation cost will be compensated by that benefit.
and use Hash.Reset. This should avoid the allocations as far as I can see...
Nevertheless Go can do the compiler optimization but for this specific use case I don't think that it's required.
Thanks for the suggestion. This is what I did. Code is available at github.com/chmike/securecookie.
My secure cookie get and set need only 3 allocations and they are all outside of the value encoding and decoding. As a comparison, the gorilla secure cookie get/set require respectively 37 and 39 allocations. While there is a factor 10 difference in number of allocations, the speed difference is only around 3.
Of course the gorilla secure cookie code is much more readable because it use the New functions and don't use arrays to avoid heap allocations.
The problem here is that sha256.New returns a hash.Hash. If it returned a *digest then the compiler would have an easier time inlining the whole thing and then seeing that nothing escapes and removing all the allocations. There was a recent CL that does the first step of devirtualizing this kind of thing. I think we should probably let the compiler optimizations continue to the point where the obvious code just doesn't allocate. That's better than adding new API that won't be (hopefully) necessary in a few releases. It's fine of course if you want to fork sha256 for your use.