net/haproxy: SNI not working when OCSP is enabled

[martinjgrunwald]

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
• I have searched the existing issues, open and closed, and I'm convinced that mine is new.
• The title contains the plugin to which this issue belongs

Describe the bug
SNI does not seem to be working in HAProxy 4.2. It did for me with HAProxy 4.1. Now it always uses the Default Certificate under Public Services -> "My HTTPS Frontend" -> SSL Offloading. When opening a website that does not match the CN of the default cert it is still used and i get a NET::ERR_CERT_COMMON_NAME_INVALID

To Reproduce
Steps to reproduce the behavior:

1. Have multiple Certificates in the ACME client
2. Select a Default Certificate under Public Services -> "My HTTPS Frontend" -> SSL Offloading
3. Access a website that does not match the CN of that default cert
4. NET::ERR_CERT_COMMON_NAME_INVALID error appears

Expected behavior
A different cert should be used

Environment
OPNsense 24.1 (amd64).
AMD EPYC 7282 16-Core Processor (4 cores, 4 threads) (VM under Proxmox)
HAProxy 4.2

[mnaiman]

I have same problem.

For those who wants back running HaProxy before fix will be issued:

1)locate in /tmp/haproxy/ssl file *.certlist
2)in that file remove all oscp suffix, leave just file on each row, save
3)SSH
killall haproxy
/usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid

HAProxy should be running fine.

This is not final solution, and any restart or save via GUI will overwrite that.
Just emergency solution to keep HAProxy running.

[martinjgrunwald]

Am I correct to assume that the problem has to do with OCSP? Could you solve this issue by re-issuing a LE cert without OSCP Must Staple, too?

[mnaiman]

On each row, this was present "ocsp-update on" probably by this commit ac824e3

I dont want to mess with certificates - need running service.
Still thanks to devs for great plugin and hard work :)

[mnaiman]

After study of source code, even more elegant solution is Settings->Global parameters->Automatic OSCP updates -> OFF
APPLY
STILL TEMPORARY FIX - oscp certificates not working in firefox, but better than nothing :)

[fraenki]

Thanks for reporting this issue. The following patch should solve this:

opnsense-patch -c plugins 404c19f6e

Be sure to hit the "Apply" button in os-haproxy in order to activate the changes after applying the patch.

(EDIT: This is a revised patch that should also address all issues mentioned below.)

[mnaiman]

Thanks, now it is working, but there is still one small glitch.

On frontend if there is Default certificate selected, and same certificate is added among certificates in field Certificates, HAProxy refuses to start with error like that

cert file looks like that

/tmp/haproxy/ssl/61562d7491ed3.pem
/tmp/haproxy/ssl/639f7a739d5d6.pem [ocsp-update on]
/tmp/haproxy/ssl/639f6f26221f5.pem [ocsp-update on]
/tmp/haproxy/ssl/615626cd5eb6c.pem [ocsp-update on]
/tmp/haproxy/ssl/615629ace2d26.pem [ocsp-update on]
/tmp/haproxy/ssl/61562a04137d3.pem [ocsp-update on]
/tmp/haproxy/ssl/61562d473b790.pem [ocsp-update on]
/tmp/haproxy/ssl/64996cbd8cbf4.pem [ocsp-update on]
/tmp/haproxy/ssl/61562d7491ed3.pem [ocsp-update on]

when not selected in certificates, cert file looks like that
/tmp/haproxy/ssl/61562d7491ed3.pem [ocsp-update on]
/tmp/haproxy/ssl/639f7a739d5d6.pem [ocsp-update on]
/tmp/haproxy/ssl/639f6f26221f5.pem [ocsp-update on]
/tmp/haproxy/ssl/615626cd5eb6c.pem [ocsp-update on]
/tmp/haproxy/ssl/615629ace2d26.pem [ocsp-update on]
/tmp/haproxy/ssl/61562a04137d3.pem [ocsp-update on]
/tmp/haproxy/ssl/61562d473b790.pem [ocsp-update on]
/tmp/haproxy/ssl/64996cbd8cbf4.pem [ocsp-update on]

First line is where HAProxy is complaining. GUI should prevent to add to Certificates that which is default (first line), or allow to not confuse user, and backend should mark both as [ocsp-update on] (or add once and mark [ocsp-update on])

[fraenki]

@mnaiman I've revised the patch, please try again.

[mnaiman]

Did I run it correctly? It fails.
opnsense-patch -c plugins 558d564

Fetched 558d564 via https://github.com/opnsense/plugins
1 out of 2 hunks failed while patching opnsense/scripts/OPNsense/HAProxy/exportCerts.php

[fraenki]

@mnaiman That's because the previous patch already modified this file. In order to be 100% sure, reinstall the plugin...

pkg install -f os-haproxy

...and apply the patch again.

[mnaiman]

Perfect, now configs are syntactical correct in both cases and HAProxy starts!

Still have problem that certificates are not stapled (firefox complaining) but need to investigate why and if it is error just my side or wider.

[mnaiman]

It seems, firefox is right

https://www.ssllabs.com/ssltest/

[fraenki]

@mnaiman Maybe third time's a charm? Please try the 3rd patch revision.

FYI, HAProxy now provides a neat way to see OCSP update status:

echo "show ssl ocsp-updates" | socat /var/run/haproxy.socket -

[Wireheadbe]

opnsense-patch -c plugins 404c19f
.. fixes firefox! Many thanks!

[mnaiman]

Yes, I confirm, that OSCP is working now. Firefox is fine.
Thanks for effort!

[TheHellSite]

@fraenki
Thanks for the patches.

The following issues are fixed for me:

• OCSP with Firefox
• SNI with multiple certificates per Frontend

The following issue is still present for me:

• HAProxy refuses to start if a self-signed certificate is configured as (default) certificate under the SSL offloading section on a (HTTPS) frontend.

[NOTICE] (81393) : haproxy version is 2.8.5-aaba8d0
[NOTICE] (81393) : path to executable is /usr/local/sbin/haproxy
[ALERT] (81393) : config : parsing [/usr/local/etc/haproxy.conf.staging:77] : 'bind 127.4.4.3:443' in section 'frontend' : 'crt-list' : error processing line 1 in file '/tmp/haproxy/ssl/646e30990fd007.26614272.certlist' : '/tmp/haproxy/ssl/6106507cef0b9.pem' has an OCSP URI and OCSP auto-update is set to 'on' but an error occurred (maybe the issuer could not be found)'.
[ALERT] (81393) : config : Error(s) found in configuration file : /usr/local/etc/haproxy.conf.staging
[ALERT] (81393) : config : Fatal errors found in configuration.

Workaround: Once the default certificate is set to "none" HAProxy accepts the config and starts fine.

[fraenki]

@TheHellSite The error suggests that there is something odd with your certificate. Please provide the output of the following commands:

cat /tmp/haproxy/ssl/646e30990fd007.26614272.certlist

cat /tmp/haproxy/ssl/6106507cef0b9.pem

[TheHellSite]

# cat /tmp/haproxy/ssl/646e30990fd007.26614272.certlist

/tmp/haproxy/ssl/635341b3c1fe0.pem [ocsp-update on]
/tmp/haproxy/ssl/62546d24641f2.pem [ocsp-update on]
/tmp/haproxy/ssl/635bcc991a877.pem [ocsp-update on]


# cat /tmp/haproxy/ssl/6106507cef0b9.pem

cat: /tmp/haproxy/ssl/6106507cef0b9.pem: No such file or directory

The default "INVALID_SNI" certificate is self-signed and used to hide the real certificates if no valid SNI is provided by the accessing client.
See Part 8 of my tutorial: https://forum.opnsense.org/index.php?topic=23339.0

[mnaiman]

If you want privacy (read your tutorial), better user that feature on frontend

[TheHellSite]

If you want privacy (read your tutorial), better user that feature on frontend

I very much like and appriciate your response!
This is even easier and cleaner than my approach. Will change my tutorial to this!

But just out of interest, I would still like to know if my issue can be fixed.

[fraenki]

has an OCSP URI and OCSP auto-update is set to 'on' but an error occurred (maybe the issuer could not be found)

I was able to reproduce this error with an arbitrary self-signed certificate. There's not much I can do about it right now. The best workaround would probably be to avoid self-signed certificates if OCSP auto updates are enabled (use ACME certificates instead).

My gut feeling is that this may be a limitation or bug in HAProxy. I've reported this to the HAProxy project to get a confirmation.

Please note that I consider the initially reported bug fixed. Everything else should be reported as NEW issues. Every bug deserves it's own issue. 😁 Thanks!

[fraenki]

has an OCSP URI and OCSP auto-update is set to 'on' but an error occurred (maybe the issuer could not be found)

The HAProxy developers confirmed that this error message is misleading. They will probably change this error message and the HAProxy behaviour in a future release.

I've implemented a workaround for this issue: os-haproxy will first check the OCSP data of every certificate before enabling OCSP updates for it. This will be available in os-haproxy 4.3.