IPsec no rekeying or creation of Child_SAs when using OpenSSL

[somova]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

In the following configuration Opnsense with OpenSSL flavor fails to start additional phase 2 tunnels of IPsec connections. According to further reports the rekeying is also affected by this issue.

• Flavor: OpenSSL 1.x
• Version: Opnsense 22.7.4 (probably all Opnsense versions are affected)

Using the LibreSSL flavor solves the problem. Due to LibreSSL is scheduled for removal from future Opnsense releases, this is no long term solution and can only be a temporary workaround.

To Reproduce

Steps to reproduce the behavior:

1. Create a new IPsec S2S connection and configure it at least with modp8192 (DH group 18) or higher for DH key exchange
2. Create multiple isolated phase 2 tunnels even with a DH group of modp8192 or higher
3. Start the first tunnel. It should come up without any problems
4. Start the second tunnel. It should fail without notice.

Expected behavior

The expected behavior is that all tunnels will come up without any problems (and re-keying works as designed).

Describe alternatives you considered

Temporary workarounds:

• Switch back to LibreSSL despite the announcement that LibreSSL is discontinued in the future
• Lower DH key exchange security by using modp4096 or lower

Relevant log files

The log file shows the following error message: "unable to install inbound and outbound IPsec SA (SAD) in kernel".

Additional context

The problem resides in OpenSSL versions 1.x. During creation of CHILD_SAs or re-keying, parameters of a called key derive function exceed a predefined buffer of 1024 bytes with the consequence that the function returns with an error. The issue has been fixed in OpenSSL 3.0.5. It looks like there is no backport available and actually OpenSSL 3.x does not seem to be ready for FreeBSD prod systems. Maybe, there are 3 different solutions:

1. Backport the patch to OpenSSL 1.x (see references)
2. Switch to OpenSSL 3.x branch (under the premise it is stable enough)
3. Extend the support for LibreSSL

References:

• unable to install inbound and outbound IPsec SA (SAD) in kernel strongswan/strongswan#1255
• openssl/openssl@20c2876
• https://www.openssl.org/docs/man1.1.1/man3/EVP_PKEY_CTX_add1_hkdf_info.html
• https://forum.opnsense.org/index.php?topic=30525.0

Environment

Not applicable here.

[fichtner]

After careful consideration adding that patch to OpenSSL 1.x will break ABI so that is a hard no from here. Likely also why this is only available in 3.x.

Cheers,
Franco

[somova]

I don't have an overview of the whole call graph of dependent functions. But, the C file looks good. Memory for the structure with the fixed sized buffer is allocated dynamically and function calls are call by reference. Which ABI do you mean?

Nevertheless, I cannot rule out that there are no issues when extending the buffer. Maybe, it is better to wait for a stable OpenSSL 3.x coming into the FreeBSD repo.

[fichtner]

I'm referring to openssl/openssl@20c2876f24d0 statement:

"Since HKDF_MAXBUF is used to size an array in the HKDF_PKEY_CTX
structure, this adds 1 kB of memory footprint to each EVP_PKEY_CTX
used for HKDF."

If this resizes an external structure linking the "fixed" OpenSSL with "unfixed" consumers a segfault is likely if the KDF ABI is being used.

[somova]

Oh, I thought this struct is only used internally and not exported. If this is not the case, indeed it will break the ABI. So, let's wait OpenSSL 3.x is integrated into stable FreeBSD. When phasing out LibreSSL there should be a limitation notice in the change log.

This ticket can be closed.