test: replace hashlib.ripemd160 with an own implementation #23716

sipa · 2021-12-08T19:19:56Z

jamesob · 2021-12-08T19:48:07Z

ACK 5b559dc, pending CI

Verified test vectors match my system's ripemd160 installation. Did not review the actual crypto.

luke-jr · 2021-12-09T02:45:30Z

test/functional/test_framework/ripemd160.py

+# Copyright (c) 2021 Pieter Wuille
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+"""Test-only pure Python RIPEMD160 implementation."""


What makes this inherently test-only?

It's not constant time.

IMO should be mentioned in the comment.

That's indeed a good warning to add in a followup.

It's not constant time.

Given that its only use in Bitcoin is ripemd160(sha256(data)), does this not make a timing attack effectively useless?

@mmgen Yes, but it's open-source code, and open-source code gets copied and can end up being adopted for other purposes.

maflcko · 2021-12-09T08:17:44Z

After merge there will be one other remaining use of new(). (git grep 'hashlib.new'). Since it is discouraged in the python docs, we should replace that with the named constructor.

Going to merge this for our CI, but I plan to review after merge.

maflcko

Looks good and given that there are tests, this is probably correct. Left a nit to show that I read the code.

maflcko · 2021-12-09T14:37:41Z

test/functional/test_framework/ripemd160.py

+    for b in range(len(data) >> 6):
+        state = compress(*state, data[64*b:64*(b+1)])
+    # Construct final blocks (with padding and size).
+    pad = b"\x80" + b"\x00" * ((119 - len(data)) & 63)


nit: using the & operator is a slightly odd way to change the sign of an integer, no?

Might be better written as (119 - len&63) & 63 ?

See also https://docs.python.org/3/library/stdtypes.html#bitwise-operations-on-integer-types

Performing these calculations with at least one extra sign extension bit in a finite two’s complement representation (a working bit-width of 1 + max(x.bit_length(), y.bit_length()) or more) is sufficient to get the same result as if there were an infinite number of sign bits.

I'm happy to change it, but "& 63" is exactly the same as "% 64" but faster, and that should be sufficient here.

Oh it is fine. I was just commenting that &63 also changes the sign from negative to positive.

For example if the size is 172, then 119-172 will be negative and &63 will make it positive.

Edit: I checked that the two do the same:

>>> all([ (119-i)&63==(119-i&63)&63 for i in range(1000000) ]) True

Yes, the result of & is only negative if both inputs are negative.

…ementation 5b559dc Swap out hashlib.ripemd160 for own implementation (Pieter Wuille) ad3e9e1 Add pure Python RIPEMD-160 (Pieter Wuille) Pull request description: Closes bitcoin#23710. ACKs for top commit: jamesob: ACK bitcoin@5b559dc, pending CI Tree-SHA512: dcd4ea2027eac572f7ab0da434b081b9a5d6b78675e559258a446b4d254b29d93c4d2cc12da4a28303543d6d99f5f2246fde4052e84af81d18e04399b137b39e

Github-Pull: bitcoin#23716 Rebased-From: ad3e9e1

Github-Pull: bitcoin#23716 Rebased-From: 5b559dc

…own implementation 5238614 Swap out hashlib.ripemd160 for own implementation (Pieter Wuille) 9a0a924 Add pure Python RIPEMD-160 (Pieter Wuille) Pull request description: Closes #23710. ACKs for top commit: jamesob: ACK bitcoin/bitcoin@5238614, pending CI Tree-SHA512: dcd4ea2027eac572f7ab0da434b081b9a5d6b78675e559258a446b4d254b29d93c4d2cc12da4a28303543d6d99f5f2246fde4052e84af81d18e04399b137b39e

Summary: OpenSSL 3 removed `ripemd160`, see [[openssl/openssl#16994 | openssl/openssl#16994]] Thus, the functional tests fail on systems with OpenSSL 3. This is a backport of [[bitcoin/bitcoin#23716 | core#23716]] Test Plan: `ninja check-functional-extended` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D11413

…ementation 5b559dc Swap out hashlib.ripemd160 for own implementation (Pieter Wuille) ad3e9e1 Add pure Python RIPEMD-160 (Pieter Wuille) Pull request description: Closes bitcoin#23710. ACKs for top commit: jamesob: ACK bitcoin@5b559dc, pending CI Tree-SHA512: dcd4ea2027eac572f7ab0da434b081b9a5d6b78675e559258a446b4d254b29d93c4d2cc12da4a28303543d6d99f5f2246fde4052e84af81d18e04399b137b39e

* Fix build of qtbase in contrib for Gcc 11.x It adds a patch with missing include <limits> in qtbase/src/tools/moc/generator.cpp * Merge bitcoin#23716: test: replace hashlib.ripemd160 with an own implementation 5b559dc Swap out hashlib.ripemd160 for own implementation (Pieter Wuille) ad3e9e1 Add pure Python RIPEMD-160 (Pieter Wuille) Pull request description: Closes bitcoin#23710. ACKs for top commit: jamesob: ACK bitcoin@5b559dc, pending CI Tree-SHA512: dcd4ea2027eac572f7ab0da434b081b9a5d6b78675e559258a446b4d254b29d93c4d2cc12da4a28303543d6d99f5f2246fde4052e84af81d18e04399b137b39e * Updates doc for Unix build: added missing dependency bison Co-authored-by: MarcoFalke <falke.marco@gmail.com>

richardkiss · 2022-05-29T22:15:36Z

Thanks for this! I've ported to python 2 and integrated into pycoin here richardkiss/pycoin@6b3e4bb

* Fix build of qtbase in contrib for Gcc 11.x It adds a patch with missing include <limits> in qtbase/src/tools/moc/generator.cpp * Merge bitcoin#23716: test: replace hashlib.ripemd160 with an own implementation 5b559dc Swap out hashlib.ripemd160 for own implementation (Pieter Wuille) ad3e9e1 Add pure Python RIPEMD-160 (Pieter Wuille) Pull request description: Closes bitcoin#23710. ACKs for top commit: jamesob: ACK bitcoin@5b559dc, pending CI Tree-SHA512: dcd4ea2027eac572f7ab0da434b081b9a5d6b78675e559258a446b4d254b29d93c4d2cc12da4a28303543d6d99f5f2246fde4052e84af81d18e04399b137b39e * Updates doc for Unix build: added missing dependency bison Co-authored-by: MarcoFalke <falke.marco@gmail.com>

Github-Pull: bitcoin#23716 Rebased-From: ad3e9e1

Github-Pull: bitcoin#23716 Rebased-From: 5b559dc

…ementation bf79f08 Swap out hashlib.ripemd160 for own implementation (Pieter Wuille) 6bfa0be Add pure Python RIPEMD-160 (Pieter Wuille) Pull request description: Backports #23716 to 0.21. Closes #25534. Top commit has no ACKs. Tree-SHA512: d2a175d781b30249b14488b818720554995a71cae1861c7443be120a01fd7828737949ece5fc9c193a154e06f4e37d373e44ec7d9dcd64f8e86c0429abe70bb6

fanquake · 2022-07-06T08:34:26Z

Backported to 0.21 in #25538. 22.0 in #25250.

3a5ec11 remove openssl fallback for ripemd160, always use pure-python implementation (windsok) 27fa0e8 Add fallback to pure-python ripemd160 (windsok) Pull request description: Newer versions of OpenSSL have removed support for `ripemd160` by default which results in an `ValueError: unsupported hash type ripemd160` exception being raised any time the `Hash160` function is used within `python-bitcoinlib`. See bitcoin/bitcoin#23710 for further details of the issue. This PR adds a fallback to a pure-python implementation of ripemd160. The code is taken from bitcoin/bitcoin#23716, and the implementation here is based on diybitcoinhardware/embit@2e293b0 Top commit has no ACKs. Tree-SHA512: f4cdbc413e262d6c9dd1692b57743c57e217f90dbf7c3ce699a8f1dfacb86ba422d7990098e623a37b2512b307800aac7ee7614d8e31e568c769899730089da8

Summary: OpenSSL 3 removed `ripemd160`, see [[openssl/openssl#16994 | openssl/openssl#16994]] Thus, the functional tests fail on systems with OpenSSL 3. This is a backport of [[bitcoin/bitcoin#23716 | core#23716]] Test Plan: `ninja check-functional-extended` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D11413

Summary: OpenSSL 3 removed `ripemd160`, see [[openssl/openssl#16994 | openssl/openssl#16994]] Thus, the functional tests fail on systems with OpenSSL 3. This is a backport of [[bitcoin/bitcoin#23716 | core#23716]] Test Plan: `ninja check-functional-extended` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D11413 Co-authored-by: Pieter Wuille <pieter@wuille.net>

…tion 8cd9155 Swap out hashlib.ripemd160 for own implementation (Pieter Wuille) cae0315 Add pure Python RIPEMD-160 (Pieter Wuille) Pull request description: OpenSSL 3.x removed support for ripemd160, which is what python's hashlib library was using to provide this feature. As such, systems that ship with or have otherwise installed OpenSSL 3.x (Ubuntu 22.04+ for example) will encounter functional test failures. We only use hashlib in functional tests specifically for ripemd160, so can just replace the dependency with a native implementation of ripemd160. Ref: bitcoin#23716, bitcoin#23710 ACKs for top commit: panleone: tACK 8cd9155 tests are locally working on my pc Liquid369: tACK 8cd9155 Tree-SHA512: 0536d064360f9cdad16bb6dcc92020f303a94bd6ce123a9d7dbbb8f366a75e09499240f1e37caaaffa5b82d6b6793a69029a4f15ca7ad99710046122efb8ff98

…e implementation 8cd9155 Swap out hashlib.ripemd160 for own implementation (Pieter Wuille) cae0315 Add pure Python RIPEMD-160 (Pieter Wuille) Pull request description: OpenSSL 3.x removed support for ripemd160, which is what python's hashlib library was using to provide this feature. As such, systems that ship with or have otherwise installed OpenSSL 3.x (Ubuntu 22.04+ for example) will encounter functional test failures. We only use hashlib in functional tests specifically for ripemd160, so can just replace the dependency with a native implementation of ripemd160. Ref: bitcoin#23716, bitcoin#23710 ACKs for top commit: panleone: tACK 8cd9155 tests are locally working on my pc Liquid369: tACK 8cd9155 Tree-SHA512: 0536d064360f9cdad16bb6dcc92020f303a94bd6ce123a9d7dbbb8f366a75e09499240f1e37caaaffa5b82d6b6793a69029a4f15ca7ad99710046122efb8ff98

Currently test runner doesn't execute the unit tests of the ripemd160 module, so add it to the list. All other framework modules that contain unit tests are included, as can be easily checked via `git grep unittest.TestCase ./test/functional/test_framework/` This is a late follow-up to PR bitcoin#23716 (commit ad3e9e1).

82e6e3c test: add ripemd160 to test framework modules list (Sebastian Falbesoner) Pull request description: Currently test runner doesn't execute the unit tests of the ripemd160 module, so add it to the list. All other framework modules that contain unit tests are included, as can be easily checked via `$ git grep unittest.TestCase ./test/functional/test_framework/` This is a late follow-up to PR #23716 (commit ad3e9e1). ACKs for top commit: MarcoFalke: lgtm ACK 82e6e3c Tree-SHA512: 10940e215f728291c7149931a356bfc42795c098bda76d760dfa68f86443a3755e1cd35cb9a8a7b2f48880beb53f3bee3842de2d74bcadd45c7b05c13ff04203

82e6e3c test: add ripemd160 to test framework modules list (Sebastian Falbesoner) Pull request description: Currently test runner doesn't execute the unit tests of the ripemd160 module, so add it to the list. All other framework modules that contain unit tests are included, as can be easily checked via `$ git grep unittest.TestCase ./test/functional/test_framework/` This is a late follow-up to PR bitcoin#23716 (commit ad3e9e1). ACKs for top commit: MarcoFalke: lgtm ACK 82e6e3c Tree-SHA512: 10940e215f728291c7149931a356bfc42795c098bda76d760dfa68f86443a3755e1cd35cb9a8a7b2f48880beb53f3bee3842de2d74bcadd45c7b05c13ff04203

Currently test runner doesn't execute the unit tests of the ripemd160 module, so add it to the list. All other framework modules that contain unit tests are included, as can be easily checked via `git grep unittest.TestCase ./test/functional/test_framework/` This is a late follow-up to PR bitcoin#23716 (commit ad3e9e1).

sipa added 2 commits December 8, 2021 14:29

Add pure Python RIPEMD-160

ad3e9e1

Swap out hashlib.ripemd160 for own implementation

5b559dc

sipa force-pushed the 202112_ripemd160 branch from cdfaa53 to 5b559dc Compare December 8, 2021 19:30

laanwj added the Tests label Dec 8, 2021

luke-jr reviewed Dec 9, 2021

View reviewed changes

maflcko merged commit 68ca867 into bitcoin:master Dec 9, 2021

kristapsk mentioned this pull request Dec 9, 2021

Potential problem with RIPEMD160 removal from newer OpenSSL versions by default Simplexum/python-bitcointx#64

Closed

dgpv mentioned this pull request Dec 9, 2021

Replace hashlib.ripemd160 with python-only implemenation Simplexum/python-bitcointx#65

Closed

maflcko reviewed Dec 9, 2021

View reviewed changes

luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Dec 14, 2021

Add pure Python RIPEMD-160

67c9c07

Github-Pull: bitcoin#23716 Rebased-From: ad3e9e1

luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Dec 14, 2021

Swap out hashlib.ripemd160 for own implementation

bd9a6fd

Github-Pull: bitcoin#23716 Rebased-From: 5b559dc

Teasel-Github mentioned this pull request May 3, 2022

ripemd160 support deprecated in OpenSSL v3 / Ubuntu 22:04 fetchai/cosmpy#144

Closed

mmgen mentioned this pull request May 3, 2022

Pycoin does not work with latest Ubuntu (22.04 Jammy) (ripemd160 has been deprecated in OpenSSL-3) richardkiss/pycoin#399

Closed

knst mentioned this pull request May 20, 2022

docs/build: Kubuntu 22.04 build fix dashpay/dash#4843

Merged

laanwj mentioned this pull request May 30, 2022

[22.x] test: replace hashlib.ripemd160 with an own implementation #25250

Merged

luislhl mentioned this pull request Jun 22, 2022

chore(ci): update base runners and small tweaks HathorNetwork/hathor-core#427

Merged

fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Jul 4, 2022

Add pure Python RIPEMD-160

6bfa0be

Github-Pull: bitcoin#23716 Rebased-From: ad3e9e1

fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Jul 4, 2022

Swap out hashlib.ripemd160 for own implementation

bf79f08

Github-Pull: bitcoin#23716 Rebased-From: 5b559dc

fanquake mentioned this pull request Jul 4, 2022

[0.21] test: replace hashlib.ripemd160 with an own implementation #25538

Merged

windsok mentioned this pull request Jul 26, 2022

Add fallback to pure-python ripemd160 petertodd/python-bitcoinlib#277

Merged

xanimo mentioned this pull request Jul 27, 2022

qa: backports pure python ripemd160 dogecoin/dogecoin#3081

Merged

psgreco mentioned this pull request Sep 7, 2022

[backport] Backport fixes from master to elements-22.x for rc6 ElementsProject/elements#1166

Merged

shohamc1 mentioned this pull request Sep 19, 2022

Replace hashlib.ripemd160 with an pure Python implementation DeFiCh/ain#1483

Merged

yashasvi-ranawat mentioned this pull request Nov 26, 2022

Unsupported hash type ripemd160 pybitcash/bitcash#118

Closed

Fuzzbawls mentioned this pull request Mar 12, 2023

[Tests] Replace hashlib.ripemd160 with native implementation PIVX-Project/PIVX#2822

Merged

MHHukiewitz mentioned this pull request Apr 26, 2023

Fix pubkey_to_address function: add native ripemd-160 hukkin/cosmospy#37

Open

theStack mentioned this pull request Apr 29, 2023

test: add ripemd160 to test framework modules list #27542

Merged

darosior mentioned this pull request May 12, 2023

hash160 may not always work darosior/python-bip380#30

Open

delta1 added a commit to delta1/elements that referenced this pull request Jun 6, 2023

Merge 68ca867 into merged_master (Bitcoin PR bitcoin/bitcoin#23716)

1413ea7

bitcoin locked and limited conversation to collaborators Jul 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: replace hashlib.ripemd160 with an own implementation #23716

test: replace hashlib.ripemd160 with an own implementation #23716

sipa commented Dec 8, 2021

jamesob commented Dec 8, 2021

luke-jr Dec 9, 2021

sipa Dec 9, 2021

luke-jr Dec 9, 2021

Sjors Dec 9, 2021

mmgen May 3, 2022

sipa May 3, 2022

maflcko commented Dec 9, 2021 •

edited

maflcko left a comment

maflcko Dec 9, 2021

sipa Dec 9, 2021

maflcko Dec 9, 2021 •

edited

sipa Dec 9, 2021

richardkiss commented May 29, 2022

fanquake commented Jul 6, 2022

test: replace hashlib.ripemd160 with an own implementation #23716

test: replace hashlib.ripemd160 with an own implementation #23716

Conversation

sipa commented Dec 8, 2021

jamesob commented Dec 8, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maflcko commented Dec 9, 2021 • edited

maflcko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maflcko Dec 9, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richardkiss commented May 29, 2022

fanquake commented Jul 6, 2022

maflcko commented Dec 9, 2021 •

edited

maflcko Dec 9, 2021 •

edited