Add TLS server cipher preference support to hMailServer 5.6.x

[SvenKiljan]

This pull request is for hMailServer 5.6.x. For lack of a better branch, I picked 5.6.8. It should be added to a new branch for a new 5.6.x version.

The added function concerns the OpenSSL option SSL_OP_CIPHER_SERVER_PREFERENCE. One existing example of the use of this option is by NGINX using the ssl_prefer_server_ciphers option. With server cipher preference (once enabled), the server will prioritize its own cipher load order instead of that of the client.

• The NCSC IT Security Guidelines for Transport Layer Security (TLS) recommends the use of server cipher preference with specific ciphers. See B2-5 for more information.
• Enabling server cipher preference is recommended by Mozilla for very old clients.
• Server cipher preference is required to fully pass some online SMTP server tests. One example is provided by internet.nl.

I enabled the function by default through the hMailServer database SQL initialization files, based on that TLS 1.0 support (which is also required by older clients) is enabled by default as well. Users can toggle this function on the SSL/TLS page of hMailServer Administrator.

All new functionality in this PR is disabled by default for backwards compatibility.

Furthermore, another OpenSSL option, SSL_OP_PRIORITIZE_CHACHA, has been added to prioritize ChaCha20-Poly1305 on the server whenever it is prioritized by the client. This is especially useful for devices that do not support AES-NI, such as mobile devices running mail clients. This does not influence clients that do not prioritize ChaCha20-Poly1305.

[RvdHout]

Hi Sven,

Not sure what you mean with the remark about lacking a better branch, the branch you picked (5.6.8) is the latest active branch in the 5.6.x series . The master branch is targeting a (upcoming?) 5.7 64-bit release, which also supports (and enables) TLS 1.0 and TLS 1.1 by default, so perhaps this pull might be suitable for that branch as well.

[SvenKiljan]

I needed this function for a production server, and 5.7 is not ready yet. That is why this PR targets 5.6.x.

What I meant with 5.6.8 is that there is no other branch that currently targets 5.6.x, which is why 5.6.8 was chosen.

You can also add this in the next 5.6.x version whenever it is released, for example when a new OpenSSL version is available.

Also note that 5.6.8 by default enables TLS 1.1 and TLS 1.2. It also has support for TLS 1.3, but it is disabled by default.

[RvdHout]

Understandable, great to see finally someone who understand the meaning of a ALPHA release (eg: the 5.7 branch) 👍

This ssl_prefer_server_ciphers option setting is only relevant when using TLS 1.0/1.1? Guess not...but not entirely clear.
If i understand the test result on internet.nl correctly this pull targets, eg: Tenminste één van je mailservers dwingt niet zijn eigen cipher-voorkeur af ('I'), right? Knowing that tested mailserver only accepts TLS 1.2 and TLS 1.3 i assume this pull has relevance their as well

Gonna check it out, i might even include it my custom 5.6.x build that is pretty widely used that i offer through the hmailserver forum (this includes some additions and changes only present in 5.7 branch)

[SvenKiljan]

It is mostly about giving the server more control in which cipher suite is chosen. This is the explanation by OpenSSL, and this explains it quite well.

I believe it also has a function for TLS 1.3, especially when reading this:

SSL_OP_PRIORITIZE_CHACHA
When SSL_OP_CIPHER_SERVER_PREFERENCE is set, temporarily reprioritize ChaCha20-Poly1305 ciphers to the top of the server cipher list if a ChaCha20-Poly1305 cipher is at the top of the client cipher list. This helps those clients (e.g. mobile) use ChaCha20-Poly1305 if that cipher is anywhere in the server cipher list; but still allows other clients to use AES and other ciphers. Requires SSL_OP_CIPHER_SERVER_PREFERENCE.

ChaCha20-Poly1305 is supported by TLS 1.2 and TLS 1.3. Furthermore, SSL_OP_PRIORITIZE_CHACHA supposedly also influences the load order of TLS 1.3. Therefore, since SSL_OP_PRIORITIZE_CHACHA requires SSL_OP_CIPHER_SERVER_PREFERENCE and influences TLS 1.3, SSL_OP_CIPHER_SERVER_PREFERENCE influences TLS 1.3.

This brings me to another point. I will add SSL_OP_PRIORITIZE_CHACHA support to this pull request, to also support ChaCha20-Poly1305 prioritization for TLS 1.2 and TLS 1.3. I believe the original cipher suite order in hMailServer does not influence TLS 1.3 ciphers. This option does, and it might be useful to prioritize it to better support mobile email clients.

[SvenKiljan]

One example of a guideline or recommendation for the use of TLS server cipher preference is found in the NCSC IT Security Guidelines for Transport Layer Security (TLS). See B2-5 for more information.

[RvdHout]

Oops, commenting on the wrong pull...sorry!

[martinknafve]

New features are no longer added to 5.6, since it's no longer under active development. Only critical bug fixes are made. More info here: https://www.hmailserver.com/state