RIPEMD160 does not work

[mouse07410]

In short: OpenSSL-3 (and master) build normally (config attached), passes all the tests, pretends that RIPEMD is there - but fail to actually use it when asked.

With 3.0.0:

$ openssl speed ripemd160
Doing rmd160 for 3s on 16 size blocks: rmd160 error!
006E861201000000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:346:Global default library context, Algorithm (RIPEMD160 : 101), Properties ()
006E861201000000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:234:
version: 3.0.0
built on: built on: Thu Oct 14 00:15:47 2021 UTC
options:bn(64,64) 
compiler: /usr/bin/clang -fPIC -arch x86_64 -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -DL_ENDIAN -DOPENSSL_PIC -D_REENTRANT -DOPENSSL_BUILDING_OPENSSL -DZLIB -DNDEBUG -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk
CPUINFO: OPENSSL_ia32cap=0x7ffef3ffffebffff:0x139efffb
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
rmd160               0.00         0.00         0.00         0.00         0.00         0.00 

With master:

$ openssl3 speed ripemd160
Doing rmd160 for 3s on 16 size blocks: rmd160 error!
004E5D0401000000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (RIPEMD160 : 97), Properties ()
004E5D0401000000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:235:
version: 3.1.0-dev
built on: Mon Nov  8 14:51:48 2021 UTC
options: bn(64,64)
compiler: clang -fPIC -arch x86_64 -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -g -DL_ENDIAN -DOPENSSL_PIC -D_REENTRANT -DOPENSSL_BUILDING_OPENSSL -DZLIB -DZLIB_SHARED
CPUINFO: OPENSSL_ia32cap=0x7ffef3ffffebffff:0x139efffb
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
rmd160               0.00         0.00         0.00         0.00         0.00         0.00 

Also, are dynamically-loaded engines no longer supported?!

$ openssl3 engine -t
(rdrand) Intel RDRAND engine
     [ available ]
(dynamic) Dynamic engine loading support
     [ unavailable ]
(gost) Reference implementation of GOST engine
     [ available ]
$ openssl3 version
OpenSSL 3.1.0-dev  (Library: OpenSSL 3.1.0-dev )
$ 

Update

My configurator did not include enable-rmd160. Re-testing with it added.

[paulidale]

Load the legacy provider by adding -provider legacy to the path. You might also need to set the provider path, also via command line option.

Yes, engines are still supported.

[mouse07410]

Load the legacy provider by adding -provider legacy to the path. You might also need to set the provider path, also via command line option.

Could you please clarify: I want RIPEMD160 be available by default, without any special command line flag. How can I do it through, e.g., build process, or, at the very least, via openssl.cnf file?

Yes, engines are still supported.

Excellent. Could you please help understanding what are the meaning and the implications of this message:

(dynamic) Dynamic engine loading support
     [ unavailable ]

and what do I need to do to make that support "available"?

Update

Adding -provider legacy helped, but I'm still getting error message:

$ openssl3 version
OpenSSL 3.1.0-dev  (Library: OpenSSL 3.1.0-dev )
$ openssl3 speed -provider legacy rmd160
Doing rmd160 for 3s on 16 size blocks: 7105946 rmd160's in 3.00s
Doing rmd160 for 3s on 64 size blocks: 5105109 rmd160's in 2.99s
Doing rmd160 for 3s on 256 size blocks: 2798150 rmd160's in 3.00s
Doing rmd160 for 3s on 1024 size blocks: 998097 rmd160's in 3.00s
Doing rmd160 for 3s on 8192 size blocks: 131755 rmd160's in 2.96s
Doing rmd160 for 3s on 16384 size blocks: 70219 rmd160's in 2.99s
00BE861101000000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (CTR-DRBG : 0), Properties (<null>)
00BE861101000000:error:12000090:random number generator:rand_new_drbg:unable to fetch drbg:crypto/rand/rand_lib.c:567:
$ 

[paulidale]

You need to use a config file along these lines:

openssl_conf = openssl_init

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

The migration guide is a good starting place for things like this.

Try loading an engine. I'm seeing the same message here and can load them okay. openssl3 list -disabled will say if engines aren't supported (it doesn't mention ENGINE if they are).

[mouse07410]

@paulidale thank you!

Taking care of openssl.cnf the way you showed above, resolved the problem.

Remaining question regarding RIPEMD160: do I need to add enable-rmd160 to ./config ......? Or is legacy provider gets built with all the associated algorithms included, unless I explicitly configure not to do that?

Re. engine:

$ openssl3 list -disabled
Disabled algorithms:
SCTP
SSL3
$ 

I suspect I don't need either of them.

I am able to use PKCS#11 engine. But I'm still concerned about that "unavailable" message - it does not look right.

[paulidale]

RIPEMD is there by default but it doesn't hurt including it in the config line. The legacy is built by default.

[cculianu]

Question to the OpenSSL devs -- reading the above.. this looks like a regression to me. Are you guys really happy with this incredibly cumbersome way to use ripemd160?

Like I didn't look at the details all I am seeing is that you need like to edit a config file or something for it to work, or something insane.

Do you feel this is a quality product for people to use? Really? I mean.. ask yourselves -- if someone handed you OpenSSL 1.1 and it "just worked" mostly.. and then they hand you OpenSSL 3.0 and you have increased complexity for no real benefit to the average user -- is this a good thing?

[paulidale]

The choice as to which algorithms should be placed into the legacy provider is extremely conservative. The algorithms there have all been considered broken for a very long time. Rather than removing them completely, the decision was made to not make them available by default.

Honestly, nobody should be using these old algorithms. How many decades do folks need to update their cryptography use?

[cculianu]

These sound like excuses. At the end of the day OpenSSL 3.0 has gotten worse for some percentage of your install base. You are ok with that it seems.

I, as a software developer, do not believe in communism. I don't see myself as some central planning komissar that "knows better" than my users. I give my users the tools they need. Like it or not, ripemd160 is used globally by millions of people each day! It is used in some mission-critical systems moving billions of dollars around. Ripemd160 is used in bitcoin, for example, and many other cryptocurrencies to hash public keys (and thus to generate a bitcoin address). Bitcoin, maybe you heard of it: It's a trilliion-dollar blockchain. They can't just hard-fork tomorrow because you don't like ripemd160.

Now, bitcoin has its own implementation of ripemd160 internally. But external toolchains in Python, for example, that interface with Bitcoin and other blockchains need easy access to ripemd160, which they get via openssl.

Now -- you guys decided to make life harder for all of those people. And for all the package manager devs out there, and for everybody else.

In this decision you have generated negative value for the world. All because of ego. You guys are not central planners, despite your liking to believe you are. You should strive to make tools that make life easier for people, not harder.

[daira]

The choice as to which algorithms should be placed into the legacy provider is extremely conservative. The algorithms there have all been considered broken for a very long time. Rather than removing them completely, the decision was made to not make them available by default.

As a cryptographer, it's news to me that RIPEMD-160 is broken. As far as I know there is no feasible attack against either preimage resistance or collision resistance less costly than the generic attacks. The best published cryptanalytic attack against CR only reaches 34 (out of 80) steps, i.e. it is nowhere near being applicable to full RIPEMD-160.

Sure, the 160-bit output size is less than preferred for new protocols, and that limits collision-resistance security to 80 bits. But RIPEMD-160 is used in a lot of places where that isn't a particular problem (for example in Bitcoin address hashing).

More to the point, disabling it in OpenSSL isn't actually helping. It's just breaking other software (including software that isn't directly using OpenSSL, for example when it's used via Python's hashlib), in ways that inconvenience people who have no ability to change or decide what algorithms are being used.

As you can see from all the pingbacks from other projects, this isn't convincing anyone to stop using RIPEMD-160 in existing protocols. They're just using workarounds or switching away from OpenSSL (to be fair, the latter isn't necessarily a bad thing).

[RBKreckel]

I'm coming here from Debian #10200695.

The choice as to which algorithms should be placed into the legacy provider is extremely conservative. The algorithms there have all been considered broken for a very long time. Rather than removing them completely, the decision was made to not make them available by default.

Aha. So RIPEMD160 is considered broken? Citation Needed!

Honestly, nobody should be using these old algorithms. How many decades do folks need to update their cryptography use?

If it was true, then MD5 and SHA1 should have been moved to the legacy provider many years ago. There are plenty of collisions for these. There is no publicly known collision for RIPEMD160.

As it stands, by moving RIPEMD160 to the legacy provider, OpenSSL breaks established crypto software with no known security weaknesses. This should be reverted.

[gmaxwell]

I'm commenting because I see some commenters invoking Bitcoin as an excuse for their comments here, I don't have any insight into OpenSSL's process.

At best RIPEMD160 provides 80 bits of collision resistance based on size alone. 2^80 work (or a small multiple) is an attack of a scale that it happening over the next decade is not entirely inconceivable, even for well funded private parties and is absolutely credible for state level attackers. Bitcoin itself has performed more than 2^94 sha256 operations -- way more than enough to find a collision in a 160 bit hash with low memory moderate communication algorithms (and sha2-256 has considerably higher area-time costs).

The Bitcoin technical community considers RIPEMD160 inadequately secure and has been moving off of it since 2017. The most recent address type doesn't use it at all, and the prior generation only used it for a restricted backwards compatible use case. Bitcoin, of course, already doesn't use libraries like openssl because consensus consistency creates specific reliability demands that OpenSSL doesn't accommodate. As a result I think the use of "bitcoin" as a justification is fairly unconvincing.

In particular, I see the parties here citing "bitcoin" are actually affiliated with under-maintained bitcoin competitors and aren't people I would speak positively about. I apologize to the openssl folks that Bitcoins past usage has inspired such rude conduct. I doubt these messages reflect the sentiments of people actually involved in Bitcoin development (they certainly don't reflect the sentiment of this former Bitcoin developer).

Sure, RIPEMD160 probably still acceptable in contexts where collisions certainly don't matter and only second-preimage security matters but determining this can be a difficult and failure prone analysis-- e.g. many people might be quick to assume Bitcoin's use of it in addresses is safe but this is obviously untrue when addresses are generated for multiparty applications (multisignature/threshold cryptography), which are widely deployed in the space. Even experts can get this wrong-- as a post up thread claims that collisions are a non-issue in Bitcoin address hashing, which is flatly untrue (but an understandable error). Downstream users are notoriously ill-equipped to make this analysis and cryptographic experts can just copy and paste a standard implementation without showing up on a github issue thread and losing their cool.

In general, OpenSSL isn't -- and really has never been-- maintained in a way that is suitable for specialized applications like consensus protocols. It's an SSL library that happened to have an under-developed interface that exposed many internals to the outside world. The historic lack of safe interfaces has resulted in many bugs. RIPEMD160 code is extremely simple and easily imported into projects (particularly because everyone lacks fancy SIMD accelerated versions anyways). SHA3 XKCP's recent debacle aside, mature hash functions are unlikely to have implementation bugs that create problems exacerbated by code duplication.

OpenSSL v3 is a new major version. Interface breaks are to be expected. Dealing with changes like this is part of the cost of developing and maintaining software. Discouraging the use of RIPEMD160 (and other older short hash functions) in most application seems like a completely prudent decision to me, particularly if it's not important for OpenSSL's primary applications. Since RIPEMD160's usage is far less widespread than other functions like MD5 and SHA1 outside of specialist applications that can provide their own functions it also seems completely reasonable to deprecate its use more rapidly than these other functions (which should also be discouraged too!).

[kanzure]

https://www.openssl.org/blog/blog/2022/10/18/rmd160-and-the-legacy-provider/ from #17722

[RBKreckel]

Bitcoin itself has performed more than 2^94 sha256 operations -- way more than enough to find a collision in a 160 bit hash with low memory moderate communication algorithms (and sha2-256 has considerably higher area-time costs).

This is misleading. A (good) 160-bit has function requires about 2^160 attempts to find a collision except if we store enough values to mount a birthday attack. Basically, a birthday attack is a time-space trade-off: If we store 2^80 sorted values and search them for a collision in each go, then we still need roughly 2^80 such attempts to find a collision. And even Bitcoin is very very far from that.

Also, the maintenance argument seems to be a pretext to justify a past mistake. The algorithm's interface is the same as any other hash function based on the Merkle-Damgård construction and as far as I can see it never failed to compile.

The decision to remove RIPEMD160 was a mistake. Before removing it, the argument has been made that it's broken – while MD5 and SHA1 were left in. Then the argument has been made that collisions have been found – without evidence. Then, the argument has been made that it is not standardized – but there's ISO/IEC 10118-3. And the argument has been made that it is not used – but there's Bitcoin.

Please stop fishing for arguments that the removal was the right thing to do. This discussion only creates fear, uncertainty, and doubt. It is perfectly okay to move away from RIPEMD160 and use more bits. But, for now, after all we know, applications are safe. (And especially so if second preimage is the sole concern.)

Alternatively: come up with an attack!