Add a PSA Crypto hashing backend (ESP-IDF 6 / Mbed TLS 4 breaks `--enable-mbed-tls`)

[odudex]

Summary

libwally's optional --enable-mbed-tls backend (HAVE_MBEDTLS_SHA256_H / HAVE_MBEDTLS_SHA512_H) uses the legacy mbedtls_sha256_* / mbedtls_sha512_* APIs and reaches into the private fields of mbedtls_sha256_context / mbedtls_sha512_context. Mbed TLS 4.0 (with cryptography split out into TF-PSA-Crypto 1.0) removes those headers from the public API, and ESP-IDF 6.0 ships Mbed TLS 4.0 with PSA Crypto as the primary cryptography interface. Consumers such as Jade (currently on ESP-IDF v5.5.4) will not be able to build libwally with hardware-accelerated hashing once they move to ESP-IDF 6.x.

This issue proposes adding a PSA Crypto hashing backend alongside the existing mbedtls one, and describes the scope so it can be discussed before a PR.

What breaks

From the TF-PSA-Crypto 1.0 migration guide:

• mbedtls/sha256.h and mbedtls/sha512.h "are no longer available" as public headers. They now live under mbedtls/private/, and the guide "strongly recommend[s] against defining MBEDTLS_DECLARE_PRIVATE_IDENTIFIERS or MBEDTLS_ALLOW_PRIVATE_ACCESS in your own application."
• mbedtls/md.h remains with reduced functionality, but mbedtls_md_hmac_xxx() is gone.
• "PSA is now the preferred interface for computing hashes and HMAC."

From the ESP-IDF 6.0 security migration guide:

• "In ESP-IDF v6.0, multiple ESP-IDF components have been migrated from using legacy Mbed TLS cryptography APIs (for example, mbedtls_sha*_*(), mbedtls_md*_*(), etc.) to using the PSA Crypto API."
• Hardware acceleration is now exposed "through PSA drivers where available". ESP-IDF master carries PSA drivers for SHA-1/256/512 (both the DMA engines and the ESP32 "parallel engine" with software fallback), including a hash_clone entry point.
• psa_crypto_init() must be called before any cryptographic operation; ESP-IDF does this automatically at startup (esp_psa_crypto_init.c, secondary init priority 104).

Note also that the current libwally mbedtls branch is already ESP-IDF specific rather than generic mbedtls: sha256.h / sha512.h include <sdkconfig.h>, <soc/soc_caps.h> and <sha256_alt.h> / <sha512_alt.h>, and hmac.c / internal.c test CONFIG_MBEDTLS_HARDWARE_SHA, MBEDTLS_SHA256_ALT and SOC_SHA_SUPPORT_PARALLEL_ENG.

Where libwally touches mbedtls today

File	What it does	PSA impact
src/ccan/ccan/crypto/sha256/sha256.h, sha256.c	struct sha256_ctx embeds mbedtls_sha256_context; sha256_init/update/done wrap mbedtls_sha256_*	Replace with psa_hash_operation_t + psa_hash_setup/update/finish/abort
src/ccan/ccan/crypto/sha512/sha512.h, sha512.c	Same for SHA-512	Same
src/hmac.c + src/hmac.inl	HMAC uses sizeof(ctx.SHA_CTX_BUFF) to get the block size, reading c.MBEDTLS_PRIVATE(buffer)	PSA operations are opaque; use the block size directly (64 / 128, PSA_HASH_BLOCK_LENGTH())
src/internal.c (sha256_midstate, wally_sha256_midstate)	Reads c.MBEDTLS_PRIVATE(state) to export the SHA-256 midstate; byte-order special case for HW engines	PSA cannot export internal state. Keep the built-in software compression function for this one call (used by Elements asset/token issuance in transaction.c)
src/tx_io.c	TXIO_CTX_CACHEABLE 0 for external hash backends because contexts cannot be copied	With PSA, psa_hash_clone() makes context caching possible again (optional improvement)
src/ccan_config.h	Maps HAVE_MBEDTLS_SHA*_H to CCAN_CRYPTO_SHA*_USE_MBEDTLS	Add HAVE_PSA_CRYPTO_H -> CCAN_CRYPTO_SHA*_USE_PSA
configure.ac, README.md	--enable-mbed-tls option and docs	Add --enable-psa-crypto; document psa_crypto_init() requirement
src/amalgamation/combined.c	Nothing extra needed; the ccan sources are already included	None

Jade's ESP-IDF component (components/libwally-core/CMakeLists.txt) enables this path with -DHAVE_MBEDTLS_SHA256_H -DHAVE_MBEDTLS_SHA512_H and PRIV_REQUIRES mbedtls, and post-build strips the sha256_init/done / sha512_init/done symbols. Consumers would switch to a single -DHAVE_PSA_CRYPTO_H.

Proposed design

1. New backend, not a replacement. Add CCAN_CRYPTO_SHA256_USE_PSA / CCAN_CRYPTO_SHA512_USE_PSA next to the OpenSSL and mbedtls branches. Keep --enable-mbed-tls working for ESP-IDF 5.x users for now, and consider deprecating it once Jade and other consumers move to ESP-IDF 6.

2. Context type. struct sha256_ctx { psa_hash_operation_t op; }.

   • sha256_init: op = PSA_HASH_OPERATION_INIT; psa_hash_setup(&op, PSA_ALG_SHA_256).
   • sha256_update: psa_hash_update.
   • sha256_done: psa_hash_finish(&op, res->u.u8, 32, &len), psa_hash_abort on failure.
   • Same for SHA-512 with PSA_ALG_SHA_512.
   • Keep the sha256() / sha512() one-shot helpers on psa_hash_compute() so the ESP driver's one-shot path (DMA) is used where available.

3. HMAC. Stop deriving the block size from the context layout. Define the pad size as 64 / 128 for the PSA backend (and ideally for all backends; it is a constant of the hash, not the context). No functional change. Moving HMAC/PBKDF2 to psa_mac_* / PSA_ALG_PBKDF2_HMAC is possible but out of scope; it needs key import per call and offers no benefit unless a platform has hardware HMAC.

4. Midstate. wally_sha256_midstate() only needs the compression function over whole blocks. With the PSA backend, compile the built-in ccan SHA-256 transform in as a private helper for this call only. Cost is a couple of KB of flash; it removes the byte-order special-casing currently keyed on MBEDTLS_SHA256_ALT / SOC_SHA_SUPPORT_PARALLEL_ENG.

5. Initialisation. Call psa_crypto_init() from wally_init() when the PSA backend is compiled in. It is idempotent, and ESP-IDF calls it itself at boot, so this only matters for non-ESP builds against TF-PSA-Crypto. Document that consumers who hash before wally_init() must call it themselves.

6. Context copying. psa_hash_clone() is part of the PSA API and is implemented by the ESP SHA drivers. tx_io.c could set TXIO_CTX_CACHEABLE 1 for PSA via a small sha256_ctx_clone() helper. Optional follow-up; the initial PR can keep caching off as it is for mbedtls today.

7. Remove ESP-IDF specifics from ccan headers. The PSA branch should not need <sdkconfig.h>, <soc/soc_caps.h> or <sha256_alt.h>; the driver selection happens inside TF-PSA-Crypto.

Estimated size

• Roughly 150 to 250 lines changed across about 9 files; no public API or ABI change (the context structs are internal).
• Mostly mechanical: the mbedtls branches being replaced are about 20 lines per hash, plus the HMAC and midstate cleanups.
• Bindings, wasm and Python are unaffected.

Things to watch

• Stack usage. psa_hash_operation_t is a union over all enabled hash contexts (the ESP driver's SHA-512 context alone is about 230 bytes) and is larger than mbedtls_sha256_context. struct sha256_ctx is used on the stack in tx_io.c, elements.c and the HMAC path. See Increased Stack Usage After Migrating to PSA Crypto (mbedTLS 4.x) Mbed-TLS/mbedtls#10537 ("Increased Stack Usage After Migrating to PSA Crypto"). Worth measuring on Jade.
• Flash footprint. Espressif reports a 3 to 6 percent increase for their own examples after the PSA move; this is a platform cost, not a libwally one.
• Testing. Nothing in libwally's CI exercises --enable-mbed-tls today. A CI job building against TF-PSA-Crypto on Linux (built from source, since distro packages still ship Mbed TLS 2.28 / 3.6) or inside the espressif/idf:v6.0 container would give the new backend real coverage and catch future PSA API churn.
• Error handling. The ccan hash API returns void; PSA functions return psa_status_t. Failures (for example the hardware engine unavailable) would have to be surfaced by leaving the context in an aborted state so sha256_done produces an error rather than silently wrong output. This needs a small design decision.

References

• ESP-IDF 6.0 security migration guide: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/migration-guides/release-6.x/6.0/security.html
• TF-PSA-Crypto 1.0 migration guide: https://github.com/Mbed-TLS/TF-PSA-Crypto/blob/development/docs/1.0-migration-guide.md
• Mbed TLS 4.0 migration guide: https://github.com/Mbed-TLS/mbedtls/blob/development/docs/4.0-migration-guide.md
• ESP-IDF PSA SHA drivers: https://github.com/espressif/esp-idf/tree/master/components/mbedtls/port/psa_driver/esp_sha
• ESPHome's equivalent migration for reference: [sha256] Migrate to PSA Crypto API for ESP-IDF 6.0 esphome/esphome#14809 (SHA-256), [hmac_sha256] Migrate to PSA Crypto MAC API for ESP-IDF 6.0 esphome/esphome#14814 (HMAC)
• Jade's libwally component: https://github.com/Blockstream/Jade/blob/master/components/libwally-core/CMakeLists.txt

I am happy to open a PR along these lines if the approach is acceptable.

[jgriffiths]

Concept ack, the above sounds reasonable/sensible.

[odudex]

Started working here: master...odudex:libwally-core:PSA-crypto

First commit has a typo fix for CMake builds that could be cherry-picked.

The branch already works fine with ESP-IDF 5.4.

I also made a "Frankenstein" Jade build with ESP-IDF 6.1, and PSA crypto improved performance on things like BIP39 derivation (nothing that would change UX). But properly bumping ESP-IDF to 6+ on Jade would require a lot of work as mbedtls hashing is used in other parts of Jade code, and this is another topic. ESP32 classic was not tested.

I'll polish the branch and submit a PR next week.

[jgriffiths]

Thanks @odudex I'll pull the cmake fix out for sure.

Please don't update the changelog with your PRs - it makes merging difficult. The changelog is updated for human consumption manually when releases are tagged.

edit: Note that we are moving to 5.5.5 with Jade 1.0.42 (soonish).

[odudex]

Please don't update the changelog with your PRs - it makes merging difficult. The changelog is updated for human consumption manually when releases are tagged.

Noted! Please don't hesitate on creating a new commit this time with this tiny fix. I can rebase later.

[jgriffiths]

Thanks, I've merged the first 2 commits here: #549 - they will be merged once internal review is complete.

I'm not expecting changes so feel free to work on top of that branch...