SSLOptions silently ignored where GnuTLS has no [priorities] SYSTEM entry (2.4.12+)

[mvysny]

Describe the bug

Since 2.4.12 (#1105) _httpTLSStart() builds the priority string as @SYSTEM,NORMAL:.... GnuTLS expands @-prefixed keywords only from the [priorities] section of the system config file; built-in level names are not consulted there, so the intended fallback to NORMAL never happens. Debian/Ubuntu ship /etc/gnutls/config with no [priorities] section (Fedora gets one from crypto-policies), so the whole string is rejected with GNUTLS_E_INVALID_REQUEST.

That error is then discarded — tls-gnutls.c:1568:

gnutls_priority_set_direct(http->tls, priority_string, NULL);

so the session silently keeps what gnutls_set_default_priority() set at line 1323, and every SSLOptions value is ignored: MinTLS*, MaxTLS*, DenyCBC, AllowRC4. Nothing is logged. SSLOptions NoSystem restores the pre-2.4.12 behaviour, which is how I found it.

Practical effect: a printer that cannot do TLS 1.3 is unreachable over ipps, and SSLOptions MaxTLS1.2 does nothing about it. Also means SSLOptions DenyCBC MinTLS1.2 in cupsd.conf is a no-op on these distros. Same symptom reported on NixOS: NixOS/nixpkgs#467175.

To Reproduce

Ubuntu 26.04, GnuTLS 3.8.12, stock /etc/gnutls/config (no [priorities]):

$ gnutls-cli -d 3 --priority '@SYSTEM,NORMAL:-VERS-TLS-ALL:+VERS-TLS1.2' --list
gnutls[2]: resolved 'SYSTEM' to '', next 'NORMAL'
gnutls[2]: resolved 'NORMAL' to '', next ''
gnutls[2]: unable to resolve @SYSTEM,NORMAL:-VERS-TLS-ALL:+VERS-TLS1.2
Syntax error at: @SYSTEM,NORMAL:-VERS-TLS-ALL:+VERS-TLS1.2

Dropping the @SYSTEM, prefix resolves fine, and so does adding SYSTEM = NORMAL under [priorities].

Expected behavior

SSLOptions is honoured, or at minimum the failure to apply it is logged.

Suggested fix: check the return value of gnutls_priority_set_direct() and retry without the @SYSTEM, prefix if it fails.

Additional context

Secondary issue, which bites even where @SYSTEM does resolve: with no MaxTLS set, CUPS appends :+VERS-TLS-ALL, and later directives win, so the policy's version restrictions are undone.

[priorities]
SYSTEM = NORMAL:-VERS-TLS1.3

@SYSTEM                                          -> TLS1.2, TLS1.1, TLS1.0
@SYSTEM,NORMAL:+VERS-TLS-ALL:-VERS-SSL3.0:...    -> ... and TLS1.3 is back

System Information

• Ubuntu 26.04
• CUPS 2.4.16 (deb) and 2.4.19 (openprinting snap), both affected
• GnuTLS 3.8.12 and 3.7.3 — _gnutls_resolve_priorities() is identical in both
• Code unchanged in master and in libcups3 master

[mvysny]

What this causes: /etc/cups.client.conf and ipptool: this is honored: SSLOptions NoSystem MinTLS1.2 MaxTLS1.2; this is completely ignored: SSLOptions MinTLS1.2 MaxTLS1.2