OpenSSL Providers in HAProxy
In order to make a full use of the information contained in this document, you need to have OpenSSLv3.0 installed, as well as an HAProxy v2.6 or more, built with SSL enabled. All the build and configuration information provided were tested on an amd64 architecture with an Ubuntu 22.04 operating system.
As described in provider(7), an OpenSSL provider is a unit of code that provides implementations for diverse algorithms. They could be used to provide new hardware-based implementation of algorithms already implemented in OpenSSL or to provide brand new algorithms altogether for instance. The provider mechanism is comparable to the engine one and it is meant to replace it in the short to medium term. It requires an important rewriting of all existing engines in order for them to be fully functioning providers though. This partially explains why not so many providers exist yet (as of June 2022) apart from the ones created by OpenSSL themselves.
This section is a quick reminder of the engine support in HAProxy, which defined what we planned on offering through the new provider API. Engines can be loaded into HAProxy thanks to the 'ssl-engine' option, to which we can specify an optional list of algorithms whose implementation should then be taken from the loaded engine. This option then allows for instance to use dedicated hardware to perform cryptographic operations, provided that you load the associated engine from the configuration. Please note that since the engine API was marked as deprecated in OpenSSL 3.0 and is bound to disappear later, the support of engines in HAProxy was disabled by default starting with version 2.6. It could be re-enabled at build time by using the USE_ENGINE option.
Three new options were added 'ssl-provider', 'ssl-provider-path' and ssl-propquery'. They are similar to the 'provider', 'provider-path' and 'propquery' options added to OpenSSL (config(5)). They allow to load any provider and to define a default property string that will be used during algorithm fetching. The following configuration snippet allows to make use of the TPM2 provider that will be described below for instance:
global
ssl-provider-path /usr/lib/x86_64-linux-gnu/ossl-modules
ssl-provider tpm2
ssl-provider default
ssl-propquery ?provider=tpm2
What this does is ask HAProxy to tell OpenSSL to load the tpm2.so shared library that can be found under the /usr/lib/x86_64-linux-gnu/ossl-modules directory, as well as OpenSSL's 'default' provider, and to use the default property string '?provider=tpm2' that tells OpenSSL to prioritize the TPM2 provider when fetching algorithm, but that not all algorithm might be implemented there (hence the '?'). See property(7) for a full explanation about property strings in OpenSSL.
With such a configuration, HAProxy acts merely as a messenger. The task of loading and using providers is entrusted to OpenSSL, and the end-of-line user -HAProxy in this case) does not have to know which provider is used when performing cryptographic operations.
For this reason, the provider configuration can also be performed directly from OpenSSL's configuration file. An equivalent configuration could be achieved by adding the following lines to your openssl.cnf file:
[openssl_init]
providers = provider_sect
alg_section = evp_properties
# List of providers to load
[provider_sect]
tpm2 = tpm2_sect
default = default_sect
[tpm2_sect]
activate = 1
[default_sect]
activate = 1
[evp_properties]
default_properties = "?provider=tpm2"
and the environment variable OPENSSL_CONF can be used to point to the openssl configuration file you want to use (in case you don't want to use this configuration for every single application that needs OpenSSL).
The TPM2 provider is an example of completed provider. It offers the funtionalities that were previously provided by an engine (TPM2 engine). It implements the cryptographic functions for Trusted Platform Module (TPM 2.0).
The way to use this provider with HAProxy requires that you first have a usable TPM setup. It needs an integrated TPM chip (unless you just want to test the provider mechanism in which case a TPM simulator could be enough). The simplest ways to know if TPM is enabled on your machine is to check if you have a /dev/tpm0 device, or to look for TPM related messages in dmesg such as the following :
$ dmesg | grep -i TPM
[ 0.000000] efi: ACPI=0xbd7fd000 ACPI 2.0=0xbd7fd014 TPMFinalLog=0xbd62b000 SMBIOS=0xb9ed8000 SMBIOS 3.0=0xb9ecb000 MEMATTR=0xb4531018 ESRT=0xb8a70000 MOKvar=0xb4536000 RNG=0xb9fc3598 TPMEventLog=0xaa3a0018
[ 0.005224] ACPI: SSDT 0x00000000B9EAC000 000651 (v02 LENOVO Tpm2Tabl 00001000 INTL 20120711)
[ 0.005227] ACPI: TPM2 0x00000000B9EAB000 000034 (v03 LENOVO TP-R13 00001150 PTEC 00000002)
[ 0.005300] ACPI: Reserving TPM2 table memory at [mem 0xb9eab000-0xb9eab033]
[ 0.426373] tpm_tis STM7308:00: 2.0 TPM (device-id 0x0, rev-id 78)
Any message such as the following in your dmesg output can be ignored, they won't prevent the TPM from working
[ 0.430409] tpm tpm0: [Firmware Bug]: TPM interrupt not working, polling instead
If you are supposed to have a TPM chip but none of the methods above yields any result, check in your BIOS configuration if TPM is indeed enabled (under Security -> Security Chip).
There might be multiple ways to make the TPM work. The way described here will rely on the tpm2-abrmd program, which will serve as an intermediate between the provider and the actual chip.
The TPM2 provider repository contains the actual provider. Its compiling and installation is pretty straightforward. You simpy need to
./bootstrap
./configure
make
make install
Abrmd stands for Access Broker and Resource Management Daemon. This daemon will communicate via dbus with the TPM2 provider. It will manage session resources and provide isolation between all the processes that make use of the TPM functionalities. The tpm2-abrmd repository should be cloned and the tpm2-abrmd to build and install it. All the information required to build it are contained in the INSTALL.md file. If you have a non-standard OpenSSL install or if you want to enable some specific options, look at the repository's files. The "standard" way is the following :
sudo useradd --system --user-group tss
./bootstrap
./configure --with-dbuspolicydir=/etc/dbus-1/system.d
make
sudo make install
Then the resource manager can be launched as user 'tss'
sudo -u tss tpm2-abrmd
There are some common errors that can be raised by tpm2-abrmd which I stumbled upon. If you have the following error
** (tpm2-abrmd:30487): CRITICAL **: 10:50:48.747: Failed to acquire DBus name com.intel.tss2.Tabrmd. UID 106 must be allowed to "own" this name. Check DBus config and check that this is running as user tss or root.
then you might need to check where the dbus configuration files are located on your system and to set the --with-dbuspolicydir option accordingly when calling the configure script.
If you already have another tpm2-abrmd instance running, then the second one will raise the following error
ERROR:tcti:src/tss2-tcti/tcti-device.c:451:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpm0: Device or resource busy
The tpm2-tools repository is an optional one that could be used as well. It provides a plethora of applications that allow to manage TPM2 keys and that communicate directly with the chip. It also provides debugging tools such as tpm2_rc_decode which should give detailed information about an error code that you could have received while performing tpm operations. During testing I sometimes managed to put my TPM chip in "DA lockout mode", a kind of security mode that I could get out of either by clearing the TPM chip contents directly from the BIOS, or by calling the tpm2_clear command (as user 'tss').
Many TPM2 simulators exist already. They allow to test a TPM setup without interacting with an actual TPM chip. The simulator suggested by the provider's documentation is IBM's one. You just need to get it from Microsoft/IBM TPM2 simulator and run the 'tpm_server' app. The abrmd should then be launched as follows
sudo -u tss tpm2-abrmd --tcti mssim:host=localhost,port=2321
The provider side of the chain will be configured the same way (and behave the same way as well).
Once your TPM2 setup is running, you can create TPM2 keys and certificate directly from openssl. The following command will create a key and self-signed certificate via the tpm2 provider:
openssl req -provider tpm2 -x509 -subj "/C=GB/CN=foo" -keyout testkey.pem -out testcert.pem
Depending on the way you installed openssl and the provider, you might need to add a -provider-path option as well. You could check that your certificate is valid by using it in an openssl server instance:
openssl s_server -provider tpm2 -provider default -propquery ?provider=tpm2 -accept 4443 -www -key testkey.pem -cert testcert.pem
curl --cacert testcert.pem --insecure https://localhost:4443/
The curl call should be successful and you should receive an html status message including information about the ciphers used and various session parameters (because of the -www option).
Once you have a usable PEM certificate (which contains the private key and certificate), you can use it in an HAProxy frontend and handshakes should work. No provider-specific option needs to be added to frontend and backend configuration lines, the proper use of the loaded provider will be completely managed by OpenSSL.
Version history:
- 1.0-20240322
COMING_SOON.20240321
Updates:
- a recent commit of pkcs11-provider is required that can load PKCS #11 private keys
by PEM file, i.e., the
crt ...option of the HAProxy configuration DSL
pkcs11-provider is a PKCS#11 provider for OpenSSL 3. It replaces
pkcs11 engine as ENGINE is deprecated. The provider is usually installed as .../ossl-modules/pkcs11.so in the sub-folder of OpenSSL's build time --prefix (note: same filename as pkcs11 engine, but different folder). Like TPM2, this is a middleware provider that requires an actual PKCS#11 provider from the HSM vendor.
PKCS#11 providers that pkcs11-provider can use include:
- NSS softoken:
/usr/lib64/libsoftokn3.so - Yubikey:
/usr/lib64/libykcs11.so - Thales Luna:
/usr/lib/libCryptoki2_64.so - remote HSM:
/usr/lib64/pkcs11/p11-kit-client.so—to access a remote HSM/smartcard exported byp11-kit server
The native naming scheme for keys is the PKCS #11 URI - RFC7512. It is sufficient to include just enough params to locate a unique key. E.g.
pkcs11:token=NSS%20Certificate%20DB;id=%01%02%03%04;type=privatepkcs11:token=NSS%20Certificate%20DB;object=cert-private-key;type=private
Tips:
- whitespace in token names are percent-encoded;
id(a byte string) is always percent-encoded - include the
type=privateparam - don't use whitespace in labels
- URI param for the PKCS #11 attribute
CKA_LABELisobject=...NOTlabel=...
Since HAProxy loads private keys from PEM files and not PKCS #11 URIs,
the key name is stored in a special PEM object with label PKCS#11 PROVIDER URI.
This object is merely a container for the PKCS #11 URI and doesn't contain any keying material.
This is similar to the PEM label TSS2 PRIVATE KEY of TPM2. A private key in PEM for pkcs11-provider
looks like this:
-----BEGIN PKCS#11 PROVIDER URI-----
MF0aGVBLQ1MjMTEgUHJvdmlkZXIgVVJJIHYxLjAMQHBrY3MxMTp0b2tlbj1OU1Ml
MjBDZXJ0ZmlpY2F0ZSUyMERCO2lkPSUwMSUwMiUwMyUwNDt0eXBlPXByaXZhdGU=
-----END PKCS#11 PROVIDER URI-----
The contents of this PEM object is a string containing the PKCS #11 URI:
0:d=0 hl=2 l= 93 cons: SEQUENCE
2:d=1 hl=2 l= 25 prim: VISIBLESTRING :PKCS#11 Provider URI v1.0
29:d=1 hl=2 l= 64 prim: UTF8STRING :pkcs11:token=NSS%20Certfiicate%20DB;id=%01%02%03%04;type=private
The current implementation of this feature requires that this be the first object
in the PEM file: either the single PEM object file mycert.crt.key / mycert.key, or the first
object in the multi-object file mycert.crt.
A tool to create such PEM stanzas can be found in this gist uri2pem.py (requires asn1crypto to be installed).
We use NSS softoken in the following example.
OpenSSL CONF file snippet
[openssl_init]
providers = provider_sect
[provider_sect]
default = default_sect
pkcs11 = pkcs11_sect
base = base_sect
[base_sect]
activate = 1
[default_sect]
activate = 1
[pkcs11_sect]
module = /usr/lib64/ossl-modules/pkcs11.so
pkcs11-module-path = /usr/lib64/libsoftokn3.so
## non-standard argument needed for NSS softoken
pkcs11-module-init-args = configDir=/home/openssl/softoken/tokens
pkcs11-module-token-pin = /home/openssl/softoken/pinfile.txt
pkcs11-module-encode-provider-uri-to-pem = true
pkcs11-module-quirks = no-deinit no-operation-state
activate = 1
Preparation:
export OPENSSL_CONF=location-to-conf-file-with-snippet
TODO: key generation example
A private key is usually created on the HSM using vendor tools or OSS tools like pkcs11-tool
or p11tool. Ensure the PKCS #11 part is working - that you can login to the token and view the key.
# NSS softoken needs to know where the directory containing
# cert9.db, key4.db is located.
# The environment variable NSS_LIB_PARAMS applies to newer versions of softoken >= 3.96,
# otherwise you will get the system token location
$ NSS_LIB_PARAMS=configDir=/home/openssl/softoken/tokens pkcs11-tool --module /usr/lib64/libsoftokn3.so \
--pin 12345678 --slot 0x2 -O
...output...
Private Key Object; RSA
label: testCert
ID: 7a8f174b4fa07c88faa8a3d8f68319b8944d1555
Usage: decrypt, sign, signRecover, unwrap
Access: sensitive, always sensitive, extractable
# Using p11-kit we create a softoken configuration with the
# non-standard x-init-reserved parameter pointing to the token directory
$ cat ~/.config/pkcs11/modules/softoken.module
module: /usr/lib64/libsoftokn3.so
x-init-reserved: configDir=/home/openssl/softoken/tokens
$ GNUTLS_PIN=12345678 p11tool --login --list-all 'pkcs11:token=NSS%20Certificate%20DB'
...lots of output...
Object 50:
URL: pkcs11:model=NSS%203;manufacturer=Mozilla%20Foundation;serial=0000000000000000;token=NSS%20Certificate%20DB;id=%7A%8F%17%4B%4F%A0%7C%88%FA%A8%A3%D8%F6%83%19%B8%94%4D%15%55;object=testCert;type=private
Type: Private key (RSA-2048)
Label: testCert
Flags: CKA_WRAP/UNWRAP; CKA_PRIVATE; CKA_EXTRACTABLE; CKA_SENSITIVE;
ID: 7a:8f:17:4b:4f:a0:7c:88:fa:a8:a3:d8:f6:83:19:b8:94:4d:15:55
# if you have an older NSS softoken without the NSS_LIB_PARAMS feature
# OpenSC pkcs11-tool can reuse this p11-kit configuration: the slot number
# will be synthesized by p11-kit
$ pkcs11-tool --module /usr/lib64/libp11-kit.so -T
Slot 1 (0x12): NSS User Private Key and Certificate Services
token label : NSS Certificate DB
token manufacturer : Mozilla Foundation
$ pkcs11-tool --module /usr/lib64/libp11-kit.so --slot 0x12 --pin 12345678 -O
...output...
Private Key Object; RSA
label: testCert
ID: 7a8f174b4fa07c88faa8a3d8f68319b8944d1555
Usage: decrypt, sign, signRecover, unwrap
Access: sensitive, always sensitive, extractable
Create a CSR for HAProxy cert:
export OPENSSL_CONF=location-to-CONF-file
# use of short URI works, as the key and token are unique in this example
openssl req -new -subj '/O=Example Corp./CN=server.example.com' -key 'pkcs11:object=testCert;type=private'
-----BEGIN CERTIFICATE REQUEST-----
MIICozCCAYsCAQAwNTEWMBQGA1UECgwNRXhhbXBsZSBDb3JwLjEbMBkGA1UEAwwS
c2VydmVyLmV4YW1wbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAuHfwZ/6llJwTkXOfN/rIm6SlfW9kRiCjOBf5Rom6GxJDtmegaCaz6IOKf9Xj
IgkTV67yIo0o1rJ2rbZA9zxm/3VktQimOHgc6+11DZXm/YSBJn6qUvQE4bJTnn6I
90L5acoSH4scx3qRB6l9BgphB/kEcvC3Iolz8Zpxnt+drX9QboVgbBYNNHiRGTTJ
404iUO25xMlceKXTBtH+mTmTQnoICJ1V2KO9o4GuUIUutlnphCzYIl3/QWFXn8pW
Nb8NsypOBU3O47dPBGJiYOb/q3jyTRxxOBoT5DDglIYZzEJFeIiUtj3Qk4Js0r0x
Aj4UV3JZIrz9+CLoIW+RK6yJrQIDAQABoCkwJwYJKoZIhvcNAQkOMRowGDAJBgNV
HRMEAjAAMAsGA1UdDwQEAwIF4DANBgkqhkiG9w0BAQsFAAOCAQEAW2LFsfxzoNKY
y7f+Bkk9xIeZPFN0yqE2cZ0+HSBvZXQcPHeFaTrTqI7fplNi9ecMkY5a668lq0yc
styVoKIhnF6LMwITVm5R27gk1WhG2gyCqzJ8vlwm34OkWHNF49ok3HYQbOk8/dDF
scnpJutNhDDye1etOpOQYKeqvZmKgZwd+3y5eZVo2UgsZDpANKd5UnWQFEQUi7ya
+WnRuc7NVxnU5rPaYMPGIbkD2dxfsfS6LfvjJFKiKEsA8b6pv7n8adlMnUiz6hIz
NT8UXOIcYOycyLUCAyWWOS8CtkrAWQL4JBHTE0C22y/CBkmc9d6rY4oVxBqSLhe0
7dNBb4godQ==
-----END CERTIFICATE REQUEST-----
# alt: store key name in PEM file
python uri2pem.py \
'pkcs11:model=NSS%203;manufacturer=Mozilla%20Foundation;serial=0000000000000000;token=NSS%20Certificate%20DB;id=%7A%8F%17%4B%4F%A0%7C%88%FA%A8%A3%D8%F6%83%19%B8%94%4D%15%55;object=testCert;type=private' \
| tee server-cert.crt.key
-----BEGIN PKCS#11 PROVIDER URI-----
MIHmGhlQS0NTIzExIFByb3ZpZGVyIFVSSSB2MS4wDIHIcGtjczExOm1vZGVsPU5T
UyUyMDM7bWFudWZhY3R1cmVyPU1vemlsbGElMjBGb3VuZGF0aW9uO3NlcmlhbD0w
MDAwMDAwMDAwMDAwMDAwO3Rva2VuPU5TUyUyMENlcnRpZmljYXRlJTIwREI7aWQ9
JTdBJThGJTE3JTRCJTRGJUEwJTdDJTg4JUZBJUE4JUEzJUQ4JUY2JTgzJTE5JUI4
JTk0JTREJTE1JTU1O29iamVjdD10ZXN0Q2VydDt0eXBlPXByaXZhdGU=
-----END PKCS#11 PROVIDER URI-----
# create CSR from key PEM
openssl req -new -subj '/O=Example Corp./CN=server.example.com' -key server-cert.crt.key
-----BEGIN CERTIFICATE REQUEST-----
MIICozCCAYsCAQAwNTEWMBQGA1UECgwNRXhhbXBsZSBDb3JwLjEbMBkGA1UEAwwS
c2VydmVyLmV4YW1wbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAuHfwZ/6llJwTkXOfN/rIm6SlfW9kRiCjOBf5Rom6GxJDtmegaCaz6IOKf9Xj
IgkTV67yIo0o1rJ2rbZA9zxm/3VktQimOHgc6+11DZXm/YSBJn6qUvQE4bJTnn6I
90L5acoSH4scx3qRB6l9BgphB/kEcvC3Iolz8Zpxnt+drX9QboVgbBYNNHiRGTTJ
404iUO25xMlceKXTBtH+mTmTQnoICJ1V2KO9o4GuUIUutlnphCzYIl3/QWFXn8pW
Nb8NsypOBU3O47dPBGJiYOb/q3jyTRxxOBoT5DDglIYZzEJFeIiUtj3Qk4Js0r0x
Aj4UV3JZIrz9+CLoIW+RK6yJrQIDAQABoCkwJwYJKoZIhvcNAQkOMRowGDAJBgNV
HRMEAjAAMAsGA1UdDwQEAwIF4DANBgkqhkiG9w0BAQsFAAOCAQEAW2LFsfxzoNKY
y7f+Bkk9xIeZPFN0yqE2cZ0+HSBvZXQcPHeFaTrTqI7fplNi9ecMkY5a668lq0yc
styVoKIhnF6LMwITVm5R27gk1WhG2gyCqzJ8vlwm34OkWHNF49ok3HYQbOk8/dDF
scnpJutNhDDye1etOpOQYKeqvZmKgZwd+3y5eZVo2UgsZDpANKd5UnWQFEQUi7ya
+WnRuc7NVxnU5rPaYMPGIbkD2dxfsfS6LfvjJFKiKEsA8b6pv7n8adlMnUiz6hIz
NT8UXOIcYOycyLUCAyWWOS8CtkrAWQL4JBHTE0C22y/CBkmc9d6rY4oVxBqSLhe0
7dNBb4godQ==
-----END CERTIFICATE REQUEST-----
Use your CA to sign and create the certificate with cert chain: server-cert.crt.
Test with OpenSSL s_server
$ openssl s_server -cert server-cert.crt -key server-cert.crt.key
Using default temp DH parameters
ACCEPT
# after client connects...
-----BEGIN SSL SESSION PARAMETERS-----
MIGDAgEBAgIDBAQCEwIEILUOpx0N3qrYeiOZ5TO/GnmXl7SinUz+zRYW+ABkamvB
BDDKiaW3gCQTTU0IctaUOFvpwkzTaDmyolIn2pXAyzYvLQQFK6GAV444REgsqAGK
lv+hBgIEZf1PdKIEAgIcIKQGBAQBAAAArgcCBQDbl4J5swMCAR0=
-----END SSL SESSION PARAMETERS-----
Shared ciphers:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-CCM:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-CCM:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES256-CCM:AES128-GCM-SHA256:AES128-CCM:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-CCM:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-CCM:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA
Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Shared Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Supported groups: x25519:secp256r1:x448:secp521r1:secp384r1:ffdhe2048:ffdhe3072:ffdhe4096:ffdhe6144:ffdhe8192
Shared groups: x25519:secp256r1:x448:secp521r1:secp384r1:ffdhe2048:ffdhe3072:ffdhe4096:ffdhe6144:ffdhe8192
CIPHER is TLS_AES_256_GCM_SHA384
This TLS version forbids renegotiation.
# in another terminal
$ openssl s_client -connect localhost:4433
Now the server cert and key file can be configured in HAProxy
- master-worker may not work as PKCS #11 providers
don't necessarily work after
fork(); this requires PKCS #11 v3 HSMs with fork safe interfaces - TODO: in a future version(3.1?) of HAProxy it may be possible to defer
key loading to the child process to avoid
fork()issues