Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace OpenSSL AES with ctaes-based version #7689

Merged
merged 12 commits into from Jun 1, 2016
Merged

Replace OpenSSL AES with ctaes-based version #7689

merged 12 commits into from Jun 1, 2016

Conversation

@sipa
Copy link
Member

sipa commented Mar 14, 2016

This is a version of #5949 with a constant-time, slow and simple AES implementation.

Performance on modern systems should be around 2-10 Mbyte/s (for short to larger messages), which is plenty for the needs of our wallet.

@sipa sipa changed the title Replace OpenSSL AES with our own constant-time version (edit of #5949) Replace OpenSSL AES with our own constant-time version Mar 14, 2016
@laanwj laanwj added this to the 0.13.0 milestone Mar 15, 2016
@@ -29,13 +29,12 @@ $(LIBLEVELDB) $(LIBMEMENV):
endif

BITCOIN_CONFIG_INCLUDES=-I$(builddir)/config
BITCOIN_INCLUDES=-I$(builddir) -I$(builddir)/obj $(BOOST_CPPFLAGS) $(LEVELDB_CPPFLAGS) $(CRYPTO_CFLAGS) $(SSL_CFLAGS)
BITCOIN_INCLUDES=-I$(builddir) -I$(builddir)/obj $(BDB_CPPFLAGS) $(BOOST_CPPFLAGS) $(LEVELDB_CPPFLAGS) $(CRYPTO_CFLAGS) $(SSL_CFLAGS)

This comment has been minimized.

Copy link
@jonasschnelli

jonasschnelli Mar 15, 2016

Member

I guess this does not break the --disable-wallet non BDB compile option? Its probably empty if BDB was not found.

This comment has been minimized.

Copy link
@theuni

theuni Mar 17, 2016

Member

@jonasschnelli yep, just empty

@jonasschnelli

This comment has been minimized.

Copy link
Member

jonasschnelli commented Mar 15, 2016

Nice work!
Short code review utACK. Will test soon.

@sipa sipa force-pushed the sipa:const_aes branch Mar 16, 2016
@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented Mar 16, 2016

Made a small change: the RijndaelSetup function now uses no modulus or division operations anymore. All tests still pass.

@sipa sipa force-pushed the sipa:const_aes branch Mar 17, 2016
@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented Mar 17, 2016

Added more comments.

@sipa sipa force-pushed the sipa:const_aes branch Mar 17, 2016
@theuni

This comment has been minimized.

Copy link
Member

theuni commented Mar 17, 2016

Thanks @sipa for the constant-time version!

@gmaxwell
gmaxwell reviewed Mar 19, 2016
View changes
src/crypto/aes.cpp Outdated
int pos = 0;

/* The first nkeywords round key words are just taken from the key directly */
for (int i = 0; i < nkeywords; i++) {

This comment has been minimized.

Copy link
@gmaxwell

gmaxwell Mar 19, 2016

Contributor

A bounds assertion on nkeywords would be helpful prior to this line (e.g. at the start of the function). The loop requires that i<8 to prevent overflow on the stack, but this is only enforced in the caller. Similarly, the round count must be limited to not overflow the state.

This comment has been minimized.

Copy link
@sipa

sipa Mar 20, 2016

Author Member

Fixed.

@sipa sipa force-pushed the sipa:const_aes branch 3 times, most recently to e208d95 Mar 20, 2016
@jonasschnelli

This comment has been minimized.

Copy link
Member

jonasschnelli commented Mar 21, 2016

Tested ACK (52e05be).
Verified test vectors, run tests on different platforms and setups.
Tested this PR with encrypted wallet.dat from master (and vice versa).

NOT tested/verified constant time behavior.

@petertodd

This comment has been minimized.

Copy link
Contributor

petertodd commented Mar 23, 2016

Concept NACK

I don't think we should be using low-level crypto primitives code developed by us that has ~zero chance of being reviewed or used by anyone other than us. I don't care how good we think we are, thats just not a good practice.

Maybe stick this in libsecp256k1 instead?

@gmaxwell

This comment has been minimized.

Copy link
Contributor

gmaxwell commented Mar 24, 2016

@petertodd the "go put it in another library" response has a verifiable history of killing useful progress here (see also continued use of the problematic and fairly scary openssl RNG), you wouldn't provide the same complaint for random "found on the internet" code that was demonstratively broken. Seems misplaced. We don't have any performance concerns for AES but in a generic library there would be performance concerns and a different construction might be called for.

@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented Mar 24, 2016

@laanwj

This comment has been minimized.

Copy link
Member

laanwj commented Mar 24, 2016

I agree with @petertodd that ideally the code should be published separately from bitcoin as well.

This doesn't need to be a generic library. We'd like this code to be self-contained (and have a special requirement here) so using OpenSSL et al is not an option, and maintaining a new generic library is a lot of work and responsibility too.

But I can understand that some people would find a specific implementation of AES just for bitcoin core as risky. (And even though it's not used in consensus, nor anything network-facing, a bug in the wallet encryption would be a big deal.)

Did anyone check by comparing their code line by line with an alternate implementation?

Yes, people have done so.

Maybe stick this in libsecp256k1 instead?

Please no, that's scope creep for libsecp256k1. You're not trying to turn secp256k1 into a generic crypto library are you?

@luke-jr

This comment has been minimized.

Copy link
Member

luke-jr commented Mar 24, 2016

There seems to be a case to make for a "Bitcoin non-consensus crypto" library with AES, SHA512, etc...

@laanwj

This comment has been minimized.

Copy link
Member

laanwj commented Mar 24, 2016

I think we should ask the question separately from where the code is, though:

Can we get any (independent, skilled) cryptographers to review this code? At least reviewing crypto code is a mostly one-time deal, after which it will (hardly) ever change.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

gmaxwell commented Mar 24, 2016

I'd already suggested sipa split out and convert to C before he posted it because this have have independent interest as is probably the smallest constant time implementation of AES I've seen, or at least the smallest that doesn't have embarrassingly bad performance-- so no objection there.

@jonasschnelli

This comment has been minimized.

Copy link
Member

jonasschnelli commented Mar 24, 2016

I'm happy to extract this PR as C code into a C89 compatible library. I have interest to use this for my SPV library project (https://github.com/libbtc/libbtc) and for a open source hardware wallet MCU codebase: https://github.com/digitalbitbox/mcu.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

gmaxwell commented Mar 30, 2016

I have had this code running on 104 cores for several days, running a test that feeds random input through encode and decode with random keys and compares it to AES-NI.

The current maximum long term rate for the wallet application of this code in the current network is roughly 7 decrypts per second of roughly 48 bytes. At the current speed of my test harness it means that I have tested the equivalent of the network's maximum rate for thirty one thousand years without finding a fault.

Mutation testing showed very very high error correlation, meaning that any error manually introduced in the software made every execution (or nearly every execution) wrong. (So far I have not found any candidate error that didn't have this effect though automated searching, though I wouldn't be totally shocked if there were one-- it's still the case that this codebase has very high error correlation).

Pieter has a new version which is a straightforward port to plain C89 with some minor cleanup and some size reductions I contributed (the code size is still larger than the smallest AES implementations I can find (which aren't constant time), but not enormously so). I'll soon update my testing harness to that code and continue. I've also now reviewed the C89 version pretty extensively.

@laanwj

This comment has been minimized.

Copy link
Member

laanwj commented Mar 30, 2016

Thanks for the thorough testing and reviewing @gmaxwell!

@btcdrak

This comment has been minimized.

Copy link
Member

btcdrak commented Mar 30, 2016

Concept ACK

1 similar comment
@paveljanik

This comment has been minimized.

Copy link
Contributor

paveljanik commented Mar 30, 2016

Concept ACK

@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented May 13, 2016

@theuni No need for that to be a blocker, though. We could move things to C as a follow-up.

@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented May 13, 2016

Ready for merging, I hope.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

gmaxwell commented May 14, 2016

I didn't see any tests that explicit test for failure with invalid padding, except perhaps in the test that makes sure it behaves the same as OpenSSL. Perhaps if the CBC mode is ported to C in a later PR that could be addressed.

utACK. Good work sipa and cfields.

@sipa sipa changed the title [WIP] Replace OpenSSL AES with ctaes-based version Replace OpenSSL AES with ctaes-based version May 14, 2016

AES256Decrypt::~AES256Decrypt()
{
memset(&ctx, 0, sizeof(ctx));

This comment has been minimized.

Copy link
@pstratem

pstratem May 17, 2016

Contributor

Won't this just be optimized out?

This comment has been minimized.

Copy link
@theuni

theuni May 17, 2016

Member

@pstratem Probably, but it doesn't hurt to leave it here. We can replace it with something stronger when we figure out what to do about OPENSSL_cleanse.

@btcdrak

This comment has been minimized.

Copy link
Member

btcdrak commented May 17, 2016

For the record, the formal peer review was made of CTAES implementation correctness. The report can be found at http://bitcoin.sipa.be/ctaes/review.zip written by Ayo Akinyele.

@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented May 25, 2016

@theuni @jonasschnelli Feel like testing/reviewing again after the update to use ctaes?

@theuni

This comment has been minimized.

Copy link
Member

theuni commented May 26, 2016

@sipa Thanks for the reminder, I'll test/ack again today. I'm not qualified to review ctaes itself, so I'll have to defer to Ayo Akinyele's review (which appears thorough).

@theuni

This comment has been minimized.

Copy link
Member

theuni commented May 27, 2016

@sipa Please grab a quick build change: theuni@723779c

After that: ACK

@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented May 27, 2016

@theuni Merged in

@sipa sipa force-pushed the sipa:const_aes branch to 723779c May 27, 2016
@theuni

This comment has been minimized.

Copy link
Member

theuni commented Jun 1, 2016

ACK 723779c

@sipa sipa merged commit 723779c into bitcoin:master Jun 1, 2016
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
sipa added a commit that referenced this pull request Jun 1, 2016
723779c build: Enumerate ctaes rather than globbing (Cory Fields)
34ed64a crypter: add tests for crypter (Cory Fields)
0a36b9a crypter: shuffle Makefile so that crypto can be used by the wallet (Cory Fields)
976f9ec crypter: add a BytesToKey clone to replace the use of openssl (Cory Fields)
9049cde crypter: hook up the new aes cbc classes (Cory Fields)
fb96831 crypter: constify encrypt/decrypt (Cory Fields)
1c391a5 crypter: fix the stored initialization vector size (Cory Fields)
daa3841 crypto: add aes cbc tests (Cory Fields)
27a212d crypto: add AES 128/256 CBC classes (Cory Fields)
6bec172 Add ctaes-based constant time AES implementation (Pieter Wuille)
a545127 Squashed 'src/crypto/ctaes/' content from commit cd3c3ac (Pieter Wuille)
codablock added a commit to codablock/dash that referenced this pull request Dec 22, 2017
723779c build: Enumerate ctaes rather than globbing (Cory Fields)
34ed64a crypter: add tests for crypter (Cory Fields)
0a36b9a crypter: shuffle Makefile so that crypto can be used by the wallet (Cory Fields)
976f9ec crypter: add a BytesToKey clone to replace the use of openssl (Cory Fields)
9049cde crypter: hook up the new aes cbc classes (Cory Fields)
fb96831 crypter: constify encrypt/decrypt (Cory Fields)
1c391a5 crypter: fix the stored initialization vector size (Cory Fields)
daa3841 crypto: add aes cbc tests (Cory Fields)
27a212d crypto: add AES 128/256 CBC classes (Cory Fields)
6bec172 Add ctaes-based constant time AES implementation (Pieter Wuille)
a545127 Squashed 'src/crypto/ctaes/' content from commit cd3c3ac (Pieter Wuille)
@str4d str4d mentioned this pull request Aug 4, 2018
zkbot added a commit to zcash/zcash that referenced this pull request Aug 4, 2018
ZIP 32 preparations

Includes Makefile changes cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#7689
- bitcoin/bitcoin#10849

Part of #3380.
zkbot added a commit to zcash/zcash that referenced this pull request Aug 4, 2018
ZIP 32 preparations

Includes Makefile changes cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#7689
- bitcoin/bitcoin#10849

Part of #3380.
zkbot added a commit to zcash/zcash that referenced this pull request Aug 5, 2018
ZIP 32 preparations

Includes Makefile changes cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#7689
- bitcoin/bitcoin#10849

Part of #3380.
zkbot added a commit to zcash/zcash that referenced this pull request Aug 5, 2018
ZIP 32 preparations

Includes Makefile changes cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#7689
- bitcoin/bitcoin#10849

Part of #3380.
zkbot added a commit to zcash/zcash that referenced this pull request Aug 5, 2018
ZIP 32 preparations

Includes Makefile changes cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#7689
- bitcoin/bitcoin#10849

Part of #3380.
andvgal added a commit to energicryptocurrency/energi that referenced this pull request Jan 6, 2019
723779c build: Enumerate ctaes rather than globbing (Cory Fields)
34ed64a crypter: add tests for crypter (Cory Fields)
0a36b9a crypter: shuffle Makefile so that crypto can be used by the wallet (Cory Fields)
976f9ec crypter: add a BytesToKey clone to replace the use of openssl (Cory Fields)
9049cde crypter: hook up the new aes cbc classes (Cory Fields)
fb96831 crypter: constify encrypt/decrypt (Cory Fields)
1c391a5 crypter: fix the stored initialization vector size (Cory Fields)
daa3841 crypto: add aes cbc tests (Cory Fields)
27a212d crypto: add AES 128/256 CBC classes (Cory Fields)
6bec172 Add ctaes-based constant time AES implementation (Pieter Wuille)
a545127 Squashed 'src/crypto/ctaes/' content from commit cd3c3ac (Pieter Wuille)
@MarkLTZ MarkLTZ mentioned this pull request Apr 30, 2019
53 of 77 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

10 participants
You can’t perform that action at this time.