Remove OpenSSL #17265

fanquake · 2019-10-26T14:14:12Z

Now that #17165 has been merged, removing our remaining OpenSSL usage is possible.

That remaining usage was a call to RAND_bytes during the ::SLOW path of ProcRand. As well as feeding output from our RNG back into OpenSSL via RAND_add during the ::SLOW and ::SLEEP paths.

Optimistically tagged for 0.20.0. Needs discussion, potentially in an upcoming weekly meeting?

Closes #12530.

jnewbery · 2019-10-26T15:00:41Z

Concept ACK!

src/random.cpp

DrahtBot · 2019-10-26T15:31:30Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#16834 (Fetch Headers over DNS by TheBlueMatt)
#16762 (Rust-based Backup over-REST block downloader by TheBlueMatt)
#15382 (util: add runCommandParseJSON by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

sipa · 2019-10-26T19:46:24Z

I think we wanted to include some of the environmental entropy sources (statistics, pid, ...) that OpenSSL uses in our own RNG state first. See #10299. I'll PR something soon.

promag · 2019-10-26T19:48:10Z

Concept ACK.

BlockMechanic · 2019-10-27T07:22:20Z

Concept ACK.

I recently ran into openssl issues here #17123, this is awesome !

laanwj · 2019-10-27T09:55:44Z

Concept and code review ACK, agree that we should ideally get #17270 in first.

practicalswift · 2019-10-27T22:02:11Z

Concept ACK

Very pleased to see OpenSSL go :)

fanquake · 2019-10-28T13:10:05Z

Fixed doc nit above and squashed some commits together. This is waiting on #17270.

Sjors · 2019-10-29T15:47:55Z

Concept ACK after #17270. This fixes #12530.

jamesob · 2019-10-29T16:03:22Z

big Concept ACK

On the ::SLOW or ::SLEEP paths, we would feed our RNG output back into OpenSSL using RAND_add. This commit removes that functionality. RAND_add(): https://www.openssl.org/docs/manmaster/man3/RAND_add.html RAND_add() mixes the num bytes at buf into the internal state of the random generator. This function will not normally be needed, as mentioned above. The randomness argument is an estimate of how much randomness is contained in buf, in bytes, and should be a number between zero and num.

On the ::SLOW path we would use OpenSSL as an additional source of random bytes. This commit removes that functionality. Note that this was always only an additional source, and that we never checked the return value RAND_bytes(): https://www.openssl.org/docs/manmaster/man3/RAND_bytes.html RAND_bytes() puts num cryptographically strong pseudo-random bytes into buf.

laanwj · 2019-11-20T10:11:10Z

it doesn't affect any of the bitcoin core binaries, so it's off topic here. Please take your question upstream.
(That said, cross-checking against other libs is generally a good idea for cryptographic libraries.)

OpenSSL is no longer required after bitcoin/bitcoin#17265. libsodium has never been required to build bitcoind on macOS.

Summary: ``` On the ::SLOW or ::SLEEP paths, we would feed our RNG output back into OpenSSL using RAND_add. This commit removes that functionality. RAND_add(): https://www.openssl.org/docs/manmaster/man3/RAND_add.html RAND_add() mixes the num bytes at buf into the internal state of the random generator. This function will not normally be needed, as mentioned above. The randomness argument is an estimate of how much randomness is contained in buf, in bytes, and should be a number between zero and num. ``` Partial backport of core [[bitcoin/bitcoin#17265 | PR17265]]: bitcoin/bitcoin@5624ab0 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6240

Summary: ``` On the ::SLOW path we would use OpenSSL as an additional source of random bytes. This commit removes that functionality. Note that this was always only an additional source, and that we never checked the return value RAND_bytes(): https://www.openssl.org/docs/manmaster/man3/RAND_bytes.html RAND_bytes() puts num cryptographically strong pseudo-random bytes into buf. ``` Partial backport of core [[bitcoin/bitcoin#17265 | PR17265]]: bitcoin/bitcoin@4fcfcc2 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6241

Summary: Partial backport of core [[bitcoin/bitcoin#17265 | PR17265]]: bitcoin/bitcoin@b49b6b0 Depends on D6240 and D6241. Test Plan: ninja all check-all Run the Gitian builds. Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6242

RdeWilde

Great 👌🏼

Summary: ``` On the ::SLOW or ::SLEEP paths, we would feed our RNG output back into OpenSSL using RAND_add. This commit removes that functionality. RAND_add(): https://www.openssl.org/docs/manmaster/man3/RAND_add.html RAND_add() mixes the num bytes at buf into the internal state of the random generator. This function will not normally be needed, as mentioned above. The randomness argument is an estimate of how much randomness is contained in buf, in bytes, and should be a number between zero and num. ``` Partial backport of core [[bitcoin/bitcoin#17265 | PR17265]]: bitcoin/bitcoin@5624ab0 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6240

Remove OpenSSL Includes changes cherry-picked from the following upstream PRs: - bitcoin/bitcoin#7095 - bitcoin/bitcoin#17165 - Only the commit removing SSL lib detection (we have long since removed the rest). - bitcoin/bitcoin#17265 - We had already migrated away from OpenSSL for randomness. - bitcoin/bitcoin#17515 - Only the second commit.

Remove OpenSSL Includes changes cherry-picked from the following upstream PRs: - bitcoin/bitcoin#7095 - bitcoin/bitcoin#11024 - bitcoin/bitcoin#17165 - Only the commit removing SSL lib detection (we have long since removed the rest). - bitcoin/bitcoin#17265 - We had already migrated away from OpenSSL for randomness. - bitcoin/bitcoin#17515 - Only the second commit. Closes #145.

e5a0bec doc: add OpenSSL removal to release-notes.md (fanquake) 397dbae ci: remove OpenSSL installation (fanquake) a4eb839 doc: remove OpenSSL from build instructions and licensing info (fanquake) 648b2e3 depends: remove OpenSSL package (fanquake) 8983ee3 build: remove OpenSSL detection and libs (fanquake) b49b6b0 random: Remove remaining OpenSSL calls and locking infrastructure (fanquake) 4fcfcc2 random: stop retrieving random bytes from OpenSSL (fanquake) 5624ab0 random: stop feeding RNG output back into OpenSSL (fanquake) Pull request description: Now that bitcoin#17165 has been merged, removing our remaining OpenSSL usage is possible. That remaining usage was a call to [`RAND_bytes`](https://www.openssl.org/docs/manmaster/man3/RAND_bytes.html) during the ::SLOW path of [ProcRand](https://github.com/bitcoin/bitcoin/blob/master/src/random.cpp#L616). As well as feeding output from our RNG back into OpenSSL via [`RAND_add`](https://www.openssl.org/docs/manmaster/man3/RAND_add.html) during the ::SLOW and ::SLEEP paths. Optimistically tagged for `0.20.0`. Needs discussion, potentially in an upcoming weekly meeting? Closes bitcoin#12530. ACKs for top commit: MarcoFalke: ACK e5a0bec laanwj: ACK e5a0bec Tree-SHA512: 02fce08ec91d20e0da51e9314eec53dcf8699cded02f0a005417d627520c20b826332cb42bdae132af283d4903aa3088a9f613f3aea915d655a51532a4d4796c

…d build ea3c7e5 test: Remove libssl-dev packages from CI scripts (Wladimir J. van der Laan) 7ea5526 test: remove lsan suppression for libcrypto (Wladimir J. van der Laan) 2d70665 build: remove libcrypto as internal dependency in libbitcoinconsensus.pc (Wladimir J. van der Laan) 278751e doc: Remove ssl as a required dependency from build-unix (Wladimir J. van der Laan) Pull request description: Some doc and build cleanups following bitcoin#17265. I intentionally left the libssl-dev install in `gitian-win-signer.yml`, as it's necessary for the ossl signer. ACKs for top commit: MarcoFalke: ACK ea3c7e5 🗯 jamesob: ACK bitcoin@ea3c7e5 practicalswift: ACK ea3c7e5 - nice! fanquake: ACK ea3c7e5 - thanks. Tree-SHA512: 67ea35bdd6d6e512d69e6734713534c88cae033a2ed695677ea15c3e3d5ff570374e342775c88e60877fa43a19047853e7b2a433e2c9a4349a5c423726a7457e

5563331 Snap: remove openssl from nightly snapcraft build requirements (Fuzzbawls) 686bfad doc: Add OpenSSL removal to release notes (Fuzzbawls) f669248 ci: remove OpenSSL installation (Fuzzbawls) 9660aec doc: remove OpenSSL from build instructions and licensing info (Fuzzbawls) 9b2e35d depends: remove OpenSSL package (Fuzzbawls) 9a81d8e CMake: remove OpenSSL detection and libs (Fuzzbawls) 53576bc build: remove OpenSSL detection and libs (fanquake) 5f30c2b Stop using OpenSSL's sha hashing in bip38 code (Fuzzbawls) d531bf2 Use our own hmac_sha256 instead of OpenSSL's in scrypt.cpp (Fuzzbawls) b687f8e Use ctaes instead of OpenSSL's AES in bip38 code (Fuzzbawls) 86c978a Remove unused openssl includes (Fuzzbawls) ab830e5 remove unused EncodeBase64Secure (Fuzzbawls) 690c938 random: Remove remaining OpenSSL calls and locking infrastructure (fanquake) 602c0b2 random: stop retrieving random bytes from OpenSSL (fanquake) b1c8396 random: stop feeding RNG output back into OpenSSL (fanquake) Pull request description: The natural follow-up to #2278, #2286, and #2288. With these three PRs merged, there are only a few minor pieces of code that still rely on OpenSSL: - a call to `RAND_bytes` during the ::SLOW path of ProcRand - feeding output from our RNG back into OpenSSL via `RAND_add` during the ::SLOW and ::SLEEP paths. - an unused function in `utilstrencodings.cpp` (`DecodeBase64Secure()`, now removed) - some stale (un-needed/un-used) header includes - bip38 exclusive usages including the following: - using OpenSSL's AES for encryption, now switched to using ctaes - using OpenSSL to do HMAC_SHA256 hashing in `crypto/scrypt.cpp`, now switched to using our native HMAC_SHA256 header - an unused function in `hash.h` (`std::string Hash(std::string input)`), now removed - a SHA256 Hash function to compute a void pointer, switched to using template objects Upstream PRs backported: bitcoin#17265, bitcoin#17515, and bitcoin#18825 The changes to bip38 were tested by doing two-way encryption/decryption between `master` and this PR ACKs for top commit: random-zebra: ACK 5563331 furszy: k, ACK 5563331 and merging.. Tree-SHA512: bfa7445d7b153bb5ea04b7b52bbedaa07ad5acd1a56221425fa5fb7c20ecbf90f392c85273734ad2a277d4fffc43b10a7a660924a8a41c175ba2fc68e6cf820f

fanquake added Needs release note Needs Conceptual Review labels Oct 26, 2019

fanquake added this to the 0.20.0 milestone Oct 26, 2019

fanquake requested a review from sipa October 26, 2019 14:14

fanquake commented Oct 26, 2019

View reviewed changes

src/random.cpp Outdated Show resolved Hide resolved

fanquake force-pushed the remove_openssl branch from f56a128 to 1ee67af Compare October 28, 2019 13:07

fanquake changed the title ~~[WIP] Remove OpenSSL~~ Remove OpenSSL Oct 28, 2019

fanquake removed the Needs Conceptual Review label Oct 28, 2019

fanquake mentioned this pull request Oct 28, 2019

Feed environment data into RNG initializers #17270

Merged

fanquake force-pushed the remove_openssl branch from 1ee67af to 24ac38d Compare November 2, 2019 13:52

fanquake added 8 commits November 18, 2019 08:48

random: Remove remaining OpenSSL calls and locking infrastructure

b49b6b0

build: remove OpenSSL detection and libs

8983ee3

depends: remove OpenSSL package

648b2e3

doc: remove OpenSSL from build instructions and licensing info

a4eb839

ci: remove OpenSSL installation

397dbae

doc: add OpenSSL removal to release-notes.md

e5a0bec

fanquake force-pushed the remove_openssl branch from 24ac38d to e5a0bec Compare November 18, 2019 14:32

fanquake marked this pull request as ready for review November 18, 2019 14:32

This was referenced Dec 2, 2019

Fixing ubuntu installation description #17651

Closed

Update build-openbsd.md #17689

Closed

emilengler mentioned this pull request Dec 15, 2019

SSL/TLS support for the HTTP(S) server #17747

Closed

daira mentioned this pull request Jan 9, 2020

Remove OpenSSL dependency. Ref: NCC-2016-007 zcash/zcash#145

Closed

fanquake added a commit to fanquake/lightning that referenced this pull request Jan 10, 2020

doc: openSSL is no longer required to build bitcoind

b07b2a9

OpenSSL is no longer required after bitcoin/bitcoin#17265. libsodium has never been required to build bitcoind on macOS.

fanquake added a commit to fanquake/lightning that referenced this pull request Jan 10, 2020

doc: openSSL is no longer required to build bitcoind

6fe72fe

OpenSSL is no longer required after bitcoin/bitcoin#17265. libsodium has never been required to build bitcoind on macOS.

fanquake mentioned this pull request Jan 10, 2020

doc: openSSL is no longer required to build bitcoind ElementsProject/lightning#3412

Merged

fanquake added a commit to fanquake/lightning that referenced this pull request Jan 11, 2020

doc: openSSL is no longer required to build bitcoind

0ee3961

OpenSSL is no longer required after bitcoin/bitcoin#17265. libsodium has never been required to build bitcoind on macOS.

cdecker pushed a commit to ElementsProject/lightning that referenced this pull request Jan 12, 2020

doc: openSSL is no longer required to build bitcoind

46f879d

OpenSSL is no longer required after bitcoin/bitcoin#17265. libsodium has never been required to build bitcoind on macOS.

azuchi mentioned this pull request Mar 27, 2020

Remove BIP70 payment server protocol from tapyrus chaintope/tapyrus-core#53

Merged

Naviabheeman mentioned this pull request Apr 1, 2020

Remove Openssl dependency from Tapyrus chaintope/tapyrus-core#61

Closed

MarkLTZ mentioned this pull request Apr 4, 2020

Bitcoin PR tracking litecoinz-core/litecoinz#14

Closed

azuchi mentioned this pull request Apr 8, 2020

Remove openssl and Improve rand chaintope/tapyrus-core#64

Merged

RdeWilde reviewed Jul 26, 2020

View reviewed changes

str4d mentioned this pull request Sep 23, 2020

Remove OpenSSL zcash/zcash#4740

Merged

Fuzzbawls mentioned this pull request Apr 29, 2021

[Refactor] Remove OpenSSL PIVX-Project/PIVX#2352

Merged

Sjors mentioned this pull request Oct 4, 2021

Edits to 2.7 Sjors/nado-book#1

Merged

UdjinM6 mentioned this pull request Feb 11, 2022

build: allow building without openssl, enables native m1 development builds dashpay/dash#4683

Merged

bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove OpenSSL #17265

Remove OpenSSL #17265

fanquake commented Oct 26, 2019 •

edited by maflcko

jnewbery commented Oct 26, 2019

DrahtBot commented Oct 26, 2019 •

edited

sipa commented Oct 26, 2019

promag commented Oct 26, 2019

BlockMechanic commented Oct 27, 2019 •

edited

laanwj commented Oct 27, 2019

practicalswift commented Oct 27, 2019

fanquake commented Oct 28, 2019

Sjors commented Oct 29, 2019

jamesob commented Oct 29, 2019

laanwj commented Nov 20, 2019

RdeWilde left a comment

Remove OpenSSL #17265

Remove OpenSSL #17265

Conversation

fanquake commented Oct 26, 2019 • edited by maflcko

jnewbery commented Oct 26, 2019

DrahtBot commented Oct 26, 2019 • edited

Conflicts

sipa commented Oct 26, 2019

promag commented Oct 26, 2019

BlockMechanic commented Oct 27, 2019 • edited

laanwj commented Oct 27, 2019

practicalswift commented Oct 27, 2019

fanquake commented Oct 28, 2019

Sjors commented Oct 29, 2019

jamesob commented Oct 29, 2019

laanwj commented Nov 20, 2019

RdeWilde left a comment

Choose a reason for hiding this comment

fanquake commented Oct 26, 2019 •

edited by maflcko

DrahtBot commented Oct 26, 2019 •

edited

BlockMechanic commented Oct 27, 2019 •

edited