Send decrypt_error alert for TLS 1.3 PSK binder failures

[Frauschi]

Description

TranslateErrorToAlert() mapped BAD_BINDER to illegal_parameter(47), which tells the peer it sent an invalid ClientHello. RFC 8446 Section 6.2 lists PSK binder validation failure under decrypt_error(51), and that is what the condition actually is: the message was well formed and only the binder MAC failed. This PR changes the mapping.

CheckPreSharedKeys() also returns BAD_BINDER when no offered PSK matched and there is no certificate to fall back to. That reuse is deliberate (commit 089f1f7c9): it keeps an unknown PSK identity indistinguishable from a failed binder. Both conditions still share one alert, so the change is indistinguishability preserving, and a comment at the return site now records the invariant at both ends.

Finally, the later certificate check in CheckPreSharedKeys() has been unreachable since 089f1f7c9 added the earlier one, so it is removed. The remaining branch falls through to the shared exit, which also restores a missing WOLFSSL_LEAVE() on the certificate fallback path.

Behaviour change

Applications inspecting wolfSSL_get_alert_history() will see alert 51 instead of 47 when a binder fails to verify, and when no PSK matched on a server with no certificate. The returned error code (BAD_BINDER) and the fatal level are unchanged.

Identity protection

Splitting BAD_BINDER off illegal_parameter exposed a second path that had to move with it. FindPsk() raised PSK_KEY_ERROR, which stays mapped to illegal_parameter, when the callback recognised an identity carrying a non-zero obfuscated_ticket_age. That ran before any binder was derived, so on a certificate-less server the two alerts would have let an unauthenticated peer enumerate valid PSK identities without holding a key.

The check is removed rather than made to fail differently. RFC 8446 Section 4.2.11: "For identities established externally, an obfuscated_ticket_age of 0 SHOULD be used, and servers MUST ignore the value." Ignoring it satisfies that requirement, closes the oracle, and lets a conformant client that sends a non-zero age complete a handshake that wolfSSL previously rejected.

test_tls13_psk_age_no_identity_oracle() guards this. It rewrites the age on the wire, since wolfSSL's own client always sends zero, and asserts a known and an unknown identity produce the same alert. A control run with an untouched ClientHello completes the handshake first, and each run asserts ssl_s->options.isPSK, which is set only after the ciphersuite match, so neither rejecting run can pass by way of a fixture that never negotiated a PSK. The accept side for a non-zero age is not covered: any on-wire rewrite invalidates the binder the client already computed, so driving it would need a hand-built ClientHello with an offline-precomputed binder.

New CI coverage

.github/workflows/psk.yml already covers NO_CERTS with TLS 1.3 PSK via static-psk-lowresource-tls13, but that config disables DH, so it only reaches the psk_ke branch. The new tls13-psk-no-certs entry in .github/configs/os-check-linux.json leaves DH enabled and covers psk_dhe_ke under NO_CERTS. There is no --disable-certs; ENABLED_CERTS is derived, so both reach it via --disable-asn.

Testing

• Full ./tests/unit.test on --enable-psk --enable-tls13: 0/1664/483/2147.
• The two updated assertions in the existing binder tests both fail if the old illegal_parameter expectation is restored, so they are not vacuous.
• A memio probe confirms the wire alert changes from 47 to 51, while wolfSSL_require_psk() still sends handshake_failure(40) and accepting controls still complete.
• The oracle is reproduced and closed: with a client patched to send a non-zero age, a known and an unknown identity gave 47 and 51 before the FindPsk() change and both give 51 after.
• Restoring only the age check, with the new test kept, fails it on two assertions: the server-side error (-333 != -423) and the alert comparison. The control run still passes there, which is what localises the failure to the age handling.
• A client patched to send a non-zero age with a valid binder is rejected with illegal_parameter(47) before the change and completes the handshake after it, confirming the RFC 8446 Section 4.2.11 behaviour.
• The removed branch was instrumented and never entered across the full suite.
• WOLFSSL_LEAVE() on the certificate fallback path: 0 before, 1 after.
• tls13-psk-no-certs passes through parallel-make-check.py with the workflow's own flags. Its make check is a genuine NO_CERTS run (0/1876/235/2111), and both test_tls13_psk_no_cert_bad_binder and the new test execute rather than skipping.

Related

Found while analysing #11077, which asked for access_denied(49) when a server PSK callback declines a valid identity. That is not actionable and #11077 should be closed as working as intended: the callback runs before the binder is verified, so nothing is authenticated when the application decides, and wc_psk_server_callback cannot distinguish an unknown identity from a denied one. A distinct alert would also reopen the enumeration oracle 089f1f7c9 closed. The mapping fixed here is the real defect that analysis surfaced.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m3

• FLASH: .text +20 B (+0.0%, 123,007 B / 262,144 B, total: 47% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text -64 B (-0.0%, 774,132 B / 1,048,576 B, total: 74% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +64 B (+0.0%, 281,344 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +64 B (+0.0%, 327,456 B / 1,048,576 B, total: 31% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text +64 B (+0.1%, 123,803 B / 262,144 B, total: 47% used)

gcc-arm-cortex-m7

• FLASH: .text +64 B (+0.0%, 201,303 B / 262,144 B, total: 77% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +64 B (+0.0%, 281,920 B / 1,048,576 B, total: 27% used)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11105

Scan targets checked: wolfssl-bugs, wolfssl-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11105

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 4
4 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11105

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.