Enable post quantum hybrid ECDHE + ML-KEM groups

[leo9800]

this patch enables post-quantum ECDHE + ML-KEM hybrid groups. (i.e. those which suffixed with -MLKEM768 or -MLKEM1024)

"store-now-decrypt-later" attack conducted by adversaries who have access to quantum computers could be mitigated by enable quantum-safe key exchange algorithms, which strengthened security for RPC-TLS.

[chucklever]

Generally speaking, the priority lists are supposed to be configured by the system (ie, the TLS library and the system's security policies). I'll try to get other reviewers to have a look at this proposal.

[leo9800]

@chucklever actually i tried adding something like

[priorities]
SYSTEM = SYSTEM = NORMAL:-GROUP-ALL:+GROUP-X25519-MLKEM768:+GROUP-X25519:+GROUP-SECP256R1:+GROUP-SECP384R1

to /etc/gnutls/config but it does not work, probably the global config has been overrided by tlshd's application config applied by gnutls_* function calls...

besides, none of the post-quantum kex groups were enabled by default as of gnutls 3.8.13, which is a fairly recent version and supports PQ indeed, while this patch enables them out-of-box.

[chucklever]

Thank you for the patch. I reviewed it against GnuTLS 3.8.13, the GnuTLS sources, and the release NEWS. The group names and version gates check out: X25519-MLKEM768 and SECP256R1-MLKEM768 arrived in 3.8.8, SECP384R1-MLKEM1024 in 3.8.9, and the resulting priority string parses and produces the intended group order. However, I found two problems, one of them fatal on common distributions.

1. The version check provides no runtime protection, and ML-KEM is a GnuTLS build option, not a version guarantee.

gnutls_check_version_numeric() short-circuits at compile time: when tlshd is built against headers >= 3.8.8, GNUTLS_VERSION_NUMBER satisfies the comparison and gnutls_check_version() is never called, so the MLKEM keywords are emitted unconditionally.

Meanwhile, the MLKEM entries in the GnuTLS group table are compiled under #ifdef HAVE_LEANCRYPTO (liboqs in 3.8.8/3.8.9). A libgnutls built without those options does not recognize the keywords, and gnutls_priority_init2() fails on an unknown group name. tlshd treats that failure as fatal: tlshd_gnutls_priority_init() returns an error and main() exits. Debian and Ubuntu build gnutls without leancrypto or liboqs (no --with-leancrypto/--with-liboqs in debian/rules), and Debian trixie ships gnutls 3.8.9, so on those systems this patch prevents tlshd from starting at all.

A runtime probe avoids both failure modes: gnutls_group_get_id("X25519-MLKEM768") != GNUTLS_GROUP_INVALID walks the same compiled-in table, so it covers both version skew and PQC-less library builds.

2. -GROUP-ALL overrides the system crypto policy.

tlshd builds its priority strings with GNUTLS_PRIORITY_INIT_DEF_APPEND so that the system default governs everything except the TLS version and the kTLS-compatible ciphers. Clearing the group list and re-adding a hardcoded set defeats that: groups an administrator disabled come back, and groups enabled by policy but absent from the hardcoded list (pure MLKEM768/MLKEM1024, anything added later) disappear.

On the platforms where tlshd is primarily deployed this is also redundant: Fedora, RHEL 10, and Oracle Linux already list the three hybrids first via crypto-policies (@SYSTEM), and SUSE is rolling out post-quantum support the same way. Administrators on distributions without crypto-policies can get the same effect with one line in /etc/gnutls/config. The remaining case, a leancrypto-enabled gnutls running with bare NORMAL defaults, is narrow and shrinking.

For these reasons tlshd should not carry a hardcoded group list. If tlshd does anything here, it would be appending the hybrid groups without -GROUP-ALL, gated on the gnutls_group_get_id() probe, so that the system policy keeps precedence and nothing is removed from it.

Minor points: these are hybrid ECDHE + ML-KEM groups rather than "EC groups", so the title and code comment should say that; and the long concat lines should wrap at 80 columns to match the rest of the file.

[leo9800]

@chucklever IIUC, the 1st issue is that the prerequisites of MLKEM hybrid group to be available it not only a GNUTLS library greater than certain version, but the lib should also be linked to leancrypto, which is identified as a lib focus on post-quantum cryptographics. (after a quick search of its name) therefore the availability of MLKEM groups should be checked via gnutls_group_get_id() but not simply gnutls_check_version_numeric() as the latter would not check if leancrypto is linked at all.

the 2nd issue is the hard-coded priority string (pstring in GNUTLS terminology) with -GROUP-ALL overriding GNUTLS' system-wide policy, now I managed to avoid use of -GROUP-ALL.

if theres no gap between the issues u mentioned and my interpret to them then the 2 commits fixed both, plus the inaccurate EC groups in the comments are also replaced by something like MLKEM + ECDH.

[chucklever]

Thanks for the quick turnaround. Issue 1 is fully resolved: gating each keyword on gnutls_group_get_id() walks the compiled-in group table, so it covers both version skew and a libgnutls built without leancrypto/liboqs. GNUTLS_GROUP_INVALID is 0, so the bare if (gnutls_group_get_id(...)) test is correct. The "tlshd won't start on Debian/Ubuntu" failure mode is gone.

Issue 2 is also addressed -- dropping -GROUP-ALL stops the patch from overriding the system crypto policy. One consequence to be aware of, though: with -GROUP-ALL gone the patch can no longer reorder groups, only add them. GnuTLS's _add_priority() (lib/priority.c) skips a group that is already present, keeping its existing position, and appends a genuinely-new group at the end of the list. So:

• On crypto-policies distributions (Fedora, RHEL 10, Oracle Linux, and SUSE as it rolls PQ out), the three hybrids are already enabled and ranked first via @SYSTEM. +GROUP-... is a duplicate and is ignored -- a no-op.
• On a bare NORMAL leancrypto build, the hybrids are appended at the end of the group list: enabled, but at lowest preference rather than prioritized.

That is the right behavior for tlshd -- it should not be reordering the administrator's group preferences -- but it means "prioritize" no longer describes what the code does. It enables the PQ hybrids where the system policy has not already. Please reconcile the title, description, and the feat: commit subject with that (the in-code comment is already fixed, thanks).

Two smaller items remain:

• The three tlshd_string_concat() lines run to 82, 85, and 86 columns. Please wrap them at 80 to match the rest of the file.
• The PR title still says "EC groups"; these are hybrid ECDHE + ML-KEM groups.

Finally, please squash the feat: + three fix: commits into a single commit before this is applied, with a message describing the final probe-gated, additive design.

[leo9800]

@chucklever PR title & desc updated, long lines truncated & wrapped and all commits squashed into one ;-D

[chucklever]

Both issues from the earlier review are resolved: gating each keyword on gnutls_group_get_id() covers version skew and libgnutls builds without leancrypto/liboqs, and dropping -GROUP-ALL keeps the system crypto policy in charge while still enabling the hybrids where policy has not. I'll tidy the changelog wording when I squash-merge. Thanks for the careful iterations.