CVE‐2026‐17507

Title: MLS membership checks compare a uint32 leaf_index as signed, admitting an out-of-range sender.

Issue affecting: BC before 1.86 (from 1.73).

Fixed versions: BC 1.86.

Platform affected: Java 8 and later.

RFC 9420 carries leaf_index as a uint32. BC's MLS implementation holds it in a signed int, so a wire value with the top bit set decodes to a negative number. That is a legitimate encoding rather than malformed input and must still decode — the MLS interop test vectors round-trip the full range — so the range has to be enforced where the index is used, not where it is read.

GroupKeySet.SecretTree.hasLeaf and Group.validateRemove instead compared the decoded value directly against the tree's (small, positive) leaf count. A signed comparison treats any negative int as less than a positive bound, so an out-of-range sender passed the membership check.

For hasLeaf that was reachable pre-authentication of the message body: the SenderData of a PrivateMessage.unprotect'd message is encrypted only under the group-shared sender_data_secret, so any current member can set it. Passing the check let the value reach LeafIndex.directPath(), whose NodeIndex.parent() arithmetic then never converges on the tree root, appending to its result list until the JVM ran out of heap. One small message from one member denied service to every other member of the group.

Both comparisons now widen with Integer.toUnsignedLong before testing, so an out-of-range sender is refused however it was encoded. Well-formed leaf indices are unaffected.

The fix was introduced in commit 1eb83471eb01.

Credit: Mirko Swillus on behalf of Alpha-Omega (alpha-omega.dev), using Scrutineer with an Anthropic Claude model provided through Project Glasswing.