Replace OpenSSL hash functions with built-in implementations #4100
Conversation
sipa
commented
Apr 27, 2014
sipa
commented
- Introduces the classes CSHA256 and CSHA512 in crypto/sha2.{cpp,h}, CSHA1 in crypto/sha1.{cpp,h} and CRIPEMD160 in crypto/ripemd160.{cpp,h}.
- The HMAC-SHA512 implementation is moved to sha2.{cpp,h} as well, with a similar interface.
- In hash.h, CHash256 (=double SHA256) and CHash160 (SHA256 + RIPEMD160) are added.
- The rest of hash.h is refactored to use those classes.
- The internal miner is also modified to use CHash256 (hiding the inner SHA256 workings).
- The getwork() RPC call is removed.
|
Benchmark of the built-in miner: old code 1111 khash/s, new code 1016 khash/s. |
|
Microbenchmark for just double-SHA256 of a 80-byte block: old 1.26us, new 1.37us |
|
IIRC, we suffer from the terribly annoying requirement that some getwork miners fail without midstate and hash1. Sure, those miners could be modified, but it seems disappointing to break an interface we want to remove anyway. The internal miner and GBT continue to see modern users, but I think it is time to retire getwork. |
|
AFAIK the only miner which needs midstate/hash1 is DiabloMiner. |
|
Agree on removing getwork completely in next major release. It doesn't scale to modern mining hardware, that alone is reason enough to get rid of it (we will have to adopt the Nit: we're ending up with a lot of hash-related implementation and header files. Maybe move them to a directory |
|
I also think we should remove getwork. We should retain an ability to mine in the stock distribution (either the integrated miner or some contrib/ solution) but that doesn't imply keeping getwork. I don't even use getwork while mining on testnet. |
|
Removed getwork(), and moved the SHA implementations to src/crypto/ (not src/hash/, as we'll also want AES there eventually, I guess). I didn't move hash.h itself there, as it doesn't contain any actual crypto code (just wrappers), and it depends on more Bitcoin-specific stuff (this way, src/crypto/ is entirely dependencyless). |
|
Good point on |
sipa
changed the title
|
Ok, bug fixed. Seems I didn't update the precomputed SHA256 state after updating the timestamp. |
sipa
changed the title
|
Added RIPEMD160 as well. |
|
untested ACK. One comment: {Read,Write}{BE,LE]{32,64} really wants to be in some sort of common header. One of my first reactions -- though outside the scope of your initial conversion -- was that the implementations of WriteBE* etc. is a "slow but safe" implementation. It should be a compiler intrinsic, as it can be on any gcc platform. And that requires a common header for such gizmos. |
|
@jeff: very preliminary test with __builtin_bswap32 results in a 4% speedup |
|
Not surprised. Enormous amounts of effort have been expended on the glibc [header] side, gcc and CPU sides of the problem, to make byte swapping faster. As a result, you will see endian.h and friends play several tricks at compile-time, and then attempt to fall back to a compiler instrinsic if that does not work. Every modern CPU has a byte-swap instruction. Any networked LE or BE platform is constantly byte-swapping. The general principle, therefore, is to include your platform's favorite endian.h, and use the macros therein. It should degenerate into a compiler intrinsic for us. In practice, this becomes painful as BSD and Linux diverge on the opinion of endian macro naming and header location. |
|
More thorough test: 6% speedup. That's probablt worth it given how many sha256'ing we do... EDIT: with __builtin_bswap32, the built-in miner is only 4% slower than the current OpenSSL-based implementation. |
|
Moved the Read/Write functions to crypto/common.h, and made them use direct read/write and/or __builtin_bswap* when possible. |
|
Ugh, endian.h doesn't exist for mingw. This will need some autotools interaction :( |
|
Small cheat: WIN32 is always little endian |
|
\o/ |
|
Rebased. |
|
Tested (linux) ACK. |
|
Extended the hash unit tests. |
|
No need to rush things, as 0.10 won't be for any time soon, but what do people think about the degree of tests here? Anything more required? |
|
I think it would be better to get this merged now so it can run on testnet |
|
General re-ACK. Comments:
|
|
Added some commits from @theuni, to build the crypto code as a separate library. |
|
@jgarzik Suggestion for the output buffer size API? I wouldn't mind a CSomething::OUTPUT_SIZE constant, but the code using it should be aware already anyway. |
|
@jgarzik Added a [Class]::OUTPUT_SIZE static constant, and use it in the method definition of Finalize and in a few other places. |
|
+1 |
|
Alternatively, I can move the non-crypto changes (just the wrapper interfaces, getwork/miner changes, unit tests, ...) to a separate pullreq, as those should be no-ops. |
|
FWIW, I announced the death of "getwork" on bitcoin-development. Nobody objected. @laanwj posted the only reply, which was a clarification for the audience. My ACK remains standing. |
|
ACK |
This also moves the HMAC-SHA512 implementation to sha2.cpp.
This allows us to include its path without making other header includes valid.
Don't depend on hard-coded platform lists
This lib has no dependencies on other bitcoin functionality. Attempting to use bitcoin headers will result in a failure to compile.
|
Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/a0495bb68c6eff9c732d458bacab10490d6452b4 for binaries and test log. |
|
Tested ACK. |
a0495bb Add <Hasher>::OUTPUT_SIZE (Pieter Wuille) 4791b99 crypto: create a separate lib for crypto functions (Cory Fields) f2647cc crypto: explicitly check for byte read/write functions (Cory Fields) 5437248 build: move bitcoin-config.h to its own directory (Cory Fields) 3820e01 Extend and move all crypto tests to crypto_tests.cpp (Pieter Wuille) 7ecd973 Move {Read,Write}{LE,BE}{32,64} to common.h and use builtins if possible (Pieter Wuille) a5bc9c0 Add built-in RIPEMD-160 implementation (Pieter Wuille) 13b5dfe Move crypto implementations to src/crypto/ (Pieter Wuille) 1cc344c Add built-in SHA-1 implementation. (Pieter Wuille) 85aab2a Switch miner.cpp to use sha2 instead of OpenSSL. (Pieter Wuille) cf0c47b Remove getwork() RPC call (Pieter Wuille) 7b4737c Switch script.cpp and hash.cpp to use sha2.cpp instead of OpenSSL. (Pieter Wuille) 977cdad Add a built-in SHA256/SHA512 implementation. (Pieter Wuille)
This adds back an implementation of getwork (together with creatework and submitwork, performing each of the two "forms" of getwork and without the need for a local wallet). This was removed in upstream Bitcoin in bitcoin/bitcoin#4100. The new version, however, is based on Namecoin's merge-mining code and thus easy to maintain. It is also necessary to support stand-alone mining with PoW data in the future out-of-the-box with existing mining software. I've tested that it works on testnet with our modified cpuminer. Proper regtests for it will be added as a follow-up.
This adds back an implementation of getwork (together with creatework and submitwork, performing each of the two "forms" of getwork and without the need for a local wallet). This was removed in upstream Bitcoin in bitcoin/bitcoin#4100. The new version, however, is based on Namecoin's merge-mining code and thus easy to maintain. It is also necessary to support stand-alone mining with PoW data in the future out-of-the-box with existing mining software. I've tested that it works on testnet with our modified cpuminer. Proper regtests for it will be added as a follow-up. Implements #33, except for the still missing regtests.
