Azure Key vault plugin sends root certificate unnecessarily

[rmalayter]

Describe the bug
win-acme Azure Key vault plugin stores root certificate in key vault, causing azure services such as App Gateway to send the root certificate unnecessarily. Only the leaf certificate and intermediate certificate should be stored in Key Vault.

To Reproduce

1. Issue any certificate and chose Azure Key Vault as the store location
2. Observe "ISRG Root X1" being transmitted as part of the handshake (example: https://www.ssllabs.com/ssltest/analyze.html?d=policy.bai.org&hideResults=on)

Expected behavior
Only the leaf certificate and intermediate certificate should be stored in Key Vault.

Log
Nothing about the root cert available in log; there are no errors and the certificate install works. It simply contains the root cert unnecessarily when it should not.

Platform:

• OS: Windows 2022, English
• Version: WACS 2.1.22.1289, 64-bit, pluggable; Azure Key vault plug-in v2.1.22.2

Additional context
Add any other context about the problem here, for example possible network issues (firewalls, proxies, NAT) that might play a role.

[WouterTinus]

The "root" certificate is kind of ill-defined with Let's Encrypt. ISRG Root X1 has two signatures: it's both self-signed (root) and also signed by DST Root CA X3, see https://letsencrypt.org/certificates/

Sending ISRG Root X1 as part of the chain allows clients that only trust DST Root CA X3 to verify the chain. We're talking really old legacy clients here. But that might be the reason for Microsoft to include it.

I'm not sure if we can influence Azure's behaviour in that sense.

[rmalayter]

Weridly, IIS on server 2019 or later doesn't send the ISRG root certificate when used as a store by WACS. I guess was expecting the "same" behavior between that an an Azure App Gateway using Key Vault.

Other than buying a cert from say DigiCert, can you think of a method by which I can test to see if this is behavior specific to the cross-signed ISRG root? Even the ECDSA ISRG Root X2 seems to be cross-signed by ISRG Root X1: https://crt.sh/?caid=183269

Basically I want to see if this is a bug in KeyVault, AppGateway, or if "pruning the root cert" is something that WACS should be doing at installation. I believe the root cert is left out automatically when "installing" WACS-managed certs for web servers like nginx or Apache, correct?

[webprofusion-chrisc]

Yeah we had the same problem in CTW, this is a feature of Azure, specifically App Gateway is not IIS, so it doesn't behave the same way and doesn't follow the Window conventions for path building. The solution is to use the ISRG Root X1 preferred chain over the Let's Encrypt default (intended for compatibility, but which windows ignores). This involves using the R3 intermediate that's signed by ISRG Root X1.

In win-acme I believe this involves setting PreferredIssuer to "ISRG Root X1" - https://www.win-acme.com/reference/settings then re-requesting your cert.

You pretty much never serve the actual Root cert (because no client should trust what you send it anyway), but I've also seen azure build a weird variation of this chain as it tries to make sense of it all.

[WouterTinus]

Chris' suggestion might work if Azure literally uses the .pfx file that's uploaded. But Microsoft products tend to be 'opinionated' on how to build chains. If it works out then I can document it for future reference.

[rmalayter]

Setting PreferredIssuer: "ISRG Root X1" results in the same behavior unfortunately. I guess I will file a bug report with Azure. You can close this one. premature information... updates below

[rmalayter]

Setting PreferredIssuer: "ISRG Root X1" results in the same behavior unfortunately. I guess I will file a bug report with Azure. You can close this one.

Ignore my last message... it seems the app gateway hasn't picked up the new certificate from the Key Vault yet. Will advise on results as soon as it changes.

[webprofusion-chrisc]

I can confirm your current certificate chain is now using the valid, modern ISRG Root X1 chain (I'm using https://chainchecker.certifytheweb.com/) to check.

[rmalayter]

Yeah we had the same problem in CTW, this is a feature of Azure, specifically App Gateway is not IIS, so it doesn't behave the same way and doesn't follow the Window conventions for path building. The solution is to use the ISRG Root X1 preferred chain over the Let's Encrypt default (intended for compatibility, but which windows ignores). This involves using the R3 intermediate that's signed by ISRG Root X1.

In win-acme I believe this involves setting PreferredIssuer to "ISRG Root X1" - https://www.win-acme.com/reference/settings then re-requesting your cert.

Wow... this actually worked! After setting PreferredIssuer: "ISRG Root X1", force-renewing the certificate (which stores it in the Azure Key vault), then waiting 30 minutes for the App Gateway to pick up the new certificate... the certifticate chain is correct! No ISRG Root X1 is being sent to the browser. Screenshot from SSL Labs test (image below).

I suppose at some point in the this should become the default issuer/chain requested by WACS, correct? Would it make sense to enter a backlog issue or roadmap item for that?

[webprofusion-chrisc]

@rmalayter it's a tricky one because:

• in the future ISRG Root X1 may not be the correct chain anymore
• if users are currently exporting certs for use with Apache/nginx or other non-windows http.sys/schannel based services, then they are actually currently serving the default [Leaf] > R3 > ISRG Root X1 > DST Root CA X3 (expired), which is the default served by let's Encrypt and intended for old Android compatibility. So, changing this default would change the compatibility of those services.

It's a quirk of Windows certificate path building that means windows clients and windows services only really serve the [Leaf] > R3 > ISRG Root X1 (self signed) chain. This meant that when the old DST Root CA X3 root expired, windows happily switched over to serving with the modern chain by itself but also at the time broke a lot of clients and there was no real way back (if serving using IIS etc).

That said, the next major release of CTW (which is unrelated to win-acme) will indeed switch to using ISRG Root X1 by default. So there is (or will be) some precedent to changing the default. I'd actually prefer if win-acme did it first so they can be a canary for any potential problems :)

[rmalayter]

makes sense... closing issue