libSceLibreSsl and libSceLibreSsl3 on a LibreSSL submodule

[simonbronk]

Gap

libSceLibreSsl (399 exports) and libSceLibreSsl3 (87) are 0% implemented; every export throws through NotImplemented_nid_no_patch. The exports are the plain OpenSSL/LibreSSL API (SHA256_Init, AES_cbc_encrypt, EVP_DigestInit_ex, SSL_read, X509_get_subject_name, ...), so the faithful implementation is LibreSSL itself built from source, the way libSceFreeType wraps the freetype submodule. No open pull request touches either library.

Which LibreSSL

Inferred from the export lists (docs/dev/debt/libSceLibreSsl*.md sources), not from Sony documentation:

• libSceLibreSsl has the pre-1.1 API (EVP_MD_CTX_create, SSL_library_init, HMAC_CTX_init) and TLS_client_method, and none of the accessors LibreSSL 2.7 added (EVP_MD_CTX_new, RSA_get0_key, X509_get0_notAfter, SSL_CTX_set_min_proto_version): LibreSSL 2.3-2.6.
• libSceLibreSsl3 adds exactly those accessors and *_up_ref: LibreSSL 2.7 or later.

Consequence: titles linked against libSceLibreSsl were compiled against headers in which EVP_MD_CTX, EVP_CIPHER_CTX, HMAC_CTX, BIO, X509 and others are not opaque. They may allocate them by value and read fields. A current LibreSSL cannot back those exports directly; they need guest-layout objects (as libSceFreeType's GuestObjects) or a 2.x build.

Questions for maintainers

1. Vendoring. A git checkout of libressl/portable does not build: autogen.sh/update.sh clone libressl/openbsd from the network and copy sources into the tree. Options:
   a. submodules 3rdparty/libressl-portable and 3rdparty/libressl-openbsd pinned to matching commits, with a CMake step that performs update.sh's copy from the local submodule (no network, works on Windows without bash);
   b. a fork of libressl/portable with the prepared tree committed, used as a single submodule;
   c. another approach you prefer.
2. Version. One modern LibreSSL (4.x) for both modules, with guest-layout wrappers where the 2.x ABI exposed struct fields; or a 2.x tag for libSceLibreSsl and a current tag for libSceLibreSsl3 (two copies, but no layout translation).
3. Coexistence with mbedTLS used by libSceSsl: no change proposed; LibreSSL symbols stay internal (--exclude-libs,ALL as libSceFreeType does).

Plan once agreed

One pull request per step, each with tests in core/libs/tests/:

1. Submodule and build; 33 primitives whose structs are caller-allocated and identical in every LibreSSL version: MD4_*, MD5_*, SHA1_*, SHA224_*, SHA256_*, SHA256, SHA384_*, SHA512_*, AES_* (8), RAND_bytes, RAND_status, RAND_add.
2. ERR_*, BN_*, CRYPTO_memcmp and the libSceLibreSsl3 one-shot digests.
3. EVP digests and ciphers, HMAC, CMAC, with the 2.x context layouts decided in question 2.
4. X509, PEM, ASN1, OCSP.
5. SSL/TLS: SSL_CTX_*, SSL_*, BIO pairs; sockets through the existing libSceNet layer; guest callbacks (SSL_CTX_set_verify, SSL_CTX_set_client_cert_cb) through SysV thunks on Windows.
6. sceLibreSslInit/InitEx/InitEx2/Term/GetMemoryPoolStats and the memory pool semantics, once a title shows how they are used.

Variadic exports (ERR_add_error_data, BIO_printf) and RAND_load_file/RAND_file_name (path translation) are listed separately and stay throwing until a title needs them.

AI-assisted: yes. I can start step 1 as soon as question 1 is settled.

[simonbronk]

To make question 1 concrete, option (a) is now implemented on a branch, with step 1 of the plan on top: https://github.com/simonbronk/AnyPS5/tree/feat/libSceLibreSsl-libressl-primitives

Vendoring (option a)

• Two submodules pinned to the matching release tags: 3rdparty/libressl-portable at v4.3.3 and 3rdparty/libressl-openbsd at libressl-v4.3.3.
• cmake/LibreSSL.cmake does what update.sh does, at configure time and without network or Bash. It copies the sources the portable Makefile.am files list into build/3rdparty/libressl, generates obj_mac.h and obj_dat.h, applies portable's patches/*.patch with git apply, and then adds portable's own CMakeLists.txt with assembly, apps, tests and install off.
• The submodules are never written to. The prepared tree is rebuilt only when either submodule HEAD or the script changes.
• No new tool requirement. objects.pl and obj_dat.pl are ported to cmake/LibreSSLObjects.py, run with Python3, which the build already needs. The port takes the license headers from the two Perl scripts in the submodule, and its output is byte-identical to Perl's for the objects of LibreSSL 2.6.5, 3.9.2 and 4.3.3.

Step 1: 33 exports

• The exports are MD4_*, MD5_*, SHA1_*, SHA224_*, SHA256_*, SHA256, SHA384_*, SHA512_*, the eight AES_* calls, RAND_bytes, RAND_status and RAND_add, linked statically into libSceLibreSsl with --exclude-libs,ALL, as libSceFreeType and libSceSsl do.
• The context structures of these calls are identical in LibreSSL 2.3.0, 2.6.5 and 4.3.3; I diffed the headers from the openbsd tags. Their 2.x sizes are asserted in Export.cpp.
• Two 2.x behaviours that 4.3.3 changed are kept:
  • SHA256(d, n, NULL) writes a static buffer, where 4.3.3 would write through the null pointer.
  • AES_set_*_key leaves the key untouched on a bad argument, where 4.3.3 writes rounds first.
• core/libs/tests/GuestLibreSsl.cpp checks:
  • the RFC 1320/1321 and FIPS 180 digest vectors;
  • the FIPS-197 AES-128/192/256 blocks;
  • the SP 800-38A ECB, CBC, CFB128 and CTR vectors, including split calls that resume through num;
  • the -1/-2 key errors;
  • that nothing writes past the 2.x structure sizes;
  • the RAND calls.
• Tested on macOS (Apple clang). The prepared tree builds libcrypto and libssl with no errors, and the test passes against that libcrypto. Linux and Windows are not built yet, and neither is the full AnyPS5 tree, which does not build on macOS.

If option (a) is acceptable, I will open it as the step 1 pull request. If you prefer (b) or something else, only cmake/LibreSSL.cmake and the submodule lines change; the exports and the test stay.

AI-assisted: yes.

[frankea]

Console data for the version and layout questions. Both libraries were loaded in a game process on a PS5 with firmware 13.40, measured 2026-10-21 with Gran Turismo 7 (PPSA01317) running. The method is ptrace remote calls from an elfldr payload:

• The payload loads the modules with the game's own sceSysmoduleLoadModule. Both 0x80000065 and 0x800000b8 return 0.
• It then calls their exports inside the game process and reads the game's memory.

Versions

• libSceLibreSsl is LibreSSL 2.5.4. SSLeay() returns 0x20000000, the constant every LibreSSL returns. The module's mapped image holds one "LibreSSL" string: "LibreSSL 2.5.4".
• libSceLibreSsl3 is LibreSSL 4.0.0. OpenSSL_version(0) returns "LibreSSL 4.0.0", and the module's image holds the same string. Types 1 to 5 return:
  • "compiler: information not available"
  • "built on: date not available"
  • "platform: information not available"
  • OPENSSLDIR: "/etc/ssl"
  • "ENGINESDIR: N/A"

So the two libraries are two LibreSSL releases, as the export names suggested. The older one being 2.5.4 pins the 2.x struct layouts titles can see.

Context sizes in libSceLibreSsl (2.5.4). Each *_init / *_Init was called on 1 KiB filled with 0xAA, and I counted the bytes it changed:

call	bytes changed	LibreSSL 2.5.4 struct
EVP_MD_CTX_init	0 to 47, all zero	EVP_MD_CTX, 48 bytes
EVP_CIPHER_CTX_init	0 to 167	EVP_CIPHER_CTX, 168 bytes
HMAC_CTX_init	0 to 151	md and the three EVP_MD_CTX. key_length and key[128] follow, and init doesn't touch them
MD5_Init	0 to 91	MD5_CTX, 92 bytes
SHA1_Init	0 to 95	SHA_CTX, 96 bytes
SHA256_Init	0 to 111	SHA256_CTX, 112 bytes
SHA512_Init	88 bytes, up to offset 215	SHA512_CTX, 216 bytes. Init leaves the 128-byte data block alone

Every size and offset matches LibreSSL 2.5.4's structs for x86-64: EVP_MD_CTX, EVP_CIPHER_CTX, HMAC_CTX and HMAC_CTX_init. The digest sizes agree with the static_asserts in #9025 (92, 96, 112 and 216), and the initial digest states are the standard ones (SHA-256's h0 = 0x6a09e667).

For the options in the post: titles that hold EVP_MD_CTX, HMAC_CTX or the digest contexts by value see these 2.5.4 sizes. A 4.x build alone, without wrappers that keep them, would not match. What the console ships is 2.5.4 for libSceLibreSsl and 4.0.0 for libSceLibreSsl3.

Not measured:

• the sceLibreSslInit*, sceLibreSslTerm and sceLibreSslGetMemoryPoolStats calls, whose signatures aren't known yet;
• whether libSceLibreSsl3 differs from upstream 4.0.0 anywhere;
• any title's actual use of either library.

AI-assisted: yes (Claude Code)

[simonbronk]

@frankea thank you, this settles the version question. What I changed on the branch:

• Submodules pinned to v4.0.0 / libressl-v4.0.0, the release libSceLibreSsl3 is. The step-1 primitives are unaffected: MD4/MD5/SHA*_CTX and AES_KEY have the same sizes in 2.5.4 and 4.0.0 (92, 92, 96, 112, 216, 244), and the static_asserts hold against the 4.0.0 headers.
• cmake/LibreSSL.cmake prepares both 4.0.0 and 4.3.3. 4.0.0 keeps opensslconf.h per architecture and has no mlkem.h. On macOS the 4.0.0 objects build (474 of 474), and so do the 4.3.3 objects.
• A bug in my script, found in a Linux run: CMake 3.28, Ubuntu 24.04's, rejects a REGEX REPLACE that can match an empty string, so configuring failed there. It now uses string(STRIP). guest_libressl passes in that container (GCC 13, ASan) on main d6eafa5.

The open question is now libSceLibreSsl's by-value 2.5.4 contexts (EVP_MD_CTX 48 bytes, EVP_CIPHER_CTX 168, HMAC_CTX), which 4.x makes opaque. I see two ways:

• (a) A second submodule pair at 2.5.4 for libSceLibreSsl. I tried the script on 2.5.4 and it doesn't fit: the headers sit elsewhere (x509v3/, no hkdf, cms, posix_time.h), so it would need its own preparation. It is also 2017 code with known CVEs, and nothing shows it builds with GCC 15 or MinGW.
• (b) One 4.0.0 build for both libraries. libSceLibreSsl keeps the 2.5.4 layouts in guest memory and holds the 4.0.0 objects behind them. EVP_MD_CTX_init would zero the 48 bytes, as you measured. EVP_DigestInit_ex would store a host context, and EVP_MD_CTX_cleanup would free it. The fields titles can read directly (digest, flags, cipher, encrypt, key_len, iv_len...) would be kept in sync.

I'd go with (b). Maintainers, does that work for you before I start on the EVP and HMAC calls?

AI-assisted: yes