SSL negotiation error when on a network that uses legacy TLS renegotiation

[johnnyreilly]

• @azure/keyvault-secrets:
• 4.7.0:
• OS X:
• nodejs
  • version:
• browser
  • name/version:
• typescript
  • version:
• Is the bug related to documentation in
  • README.md
  • source code documentation
  • SDK API docs on https://docs.microsoft.com

Describe the bug
With Node.js 18, legacy SSL support was disabled by default. As a consequence, when a library makes use of legacy SSL, a message like this presents:

EPROTO B8150000:error:0A000152:SSL routines:final_renegotiate:unsafe legacy renegotiation disabled

For some reason, it seems that the Azure KeyVault uses legacy TLS, and the consequence of that is that when you try to use the @azure/keyvault-secrets package with Node.js 18, it errors out thusly:

node_modules/@azure/core-rest-pipeline/dist/index.js:1699
                reject(new RestError(err.message, { code: (_a = err.code) !== null && _a !== void 0 ? _a : RestError.REQUEST_SEND_ERROR, request }));
                       ^

RestError: write EPROTO 40736C4DF87F0000:error:0A000152:SSL routines:final_renegotiate:unsafe legacy renegotiation disabled:../deps/openssl/openssl/ssl/statem/extensions.c:922:

    at ClientRequest.<anonymous>

To Reproduce
Steps to reproduce the behavior:

    if (args.keyVaultName) {
        const KEYVAULT_URI = `https://${args.keyVaultName}.vault.azure.net/`;
        const secretClient = new SecretClient(KEYVAULT_URI, credentials);

        console.log("Listing secrets from KeyVault:");

        // WILL ERROR HERE
        for await (const secretProperties of secretClient.listPropertiesOfSecrets()) {
            console.log("Secret properties: ", secretProperties);
        }
    }

Expected behavior
API calls work.

Screenshots
N/A

Additional context
It's possible to work around the problem like so: (this is based upon this blog post: https://johnnyreilly.com/node-18-axios-and-unsafe-legacy-renegotiation-disabled which in turn came from this Stack Overflow entry: https://stackoverflow.com/questions/74324019/allow-legacy-renegotiation-for-nodejs/77335383#77335383 )

import { createDefaultHttpClient, type HttpClient } from "@azure/core-rest-pipeline";

const httpClient = createDefaultHttpClient();
const sslLegacyAgent = new https.Agent({
    // for self signed you could also add
    // rejectUnauthorized: false,
    // allow legacy server
    secureOptions: crypto.constants.SSL_OP_LEGACY_SERVER_CONNECT,
});
const customHttpClient: HttpClient = {
    sendRequest: async (request) => {
        request.agent = sslLegacyAgent
        const response = await httpClient.sendRequest(request);
        return response;
    }
};

//....

    if (args.keyVaultName) {
        const KEYVAULT_URI = `https://${args.keyVaultName}.vault.azure.net/`;
        const secretClient = new SecretClient(KEYVAULT_URI, credentials, {
            httpClient: customHttpClient,
        });

        console.log("Listing secrets from KeyVault:");

        for await (const secretProperties of secretClient.listPropertiesOfSecrets()) {
            console.log("Secret properties: ", secretProperties);
        }
    }

[xirzec]

Hi @johnnyreilly, thanks for reporting this. Another possible workaround would be to set the agent property on the request in a custom pipeline policy (using the additionalPolicies option when configuring the client.)

At the same time, it feels odd that the KeyVault service is using legacy negotiation. Strangely, our internal test matrix that includes Node 18 and 20 doesn't seem to be impacted by this issue. @timovv are you able to reproduce this behavior on Node18?

[timovv]

This is strange. I use Node 18 on a daily basis and have never run into this issue. I tried just now running the example provided and it works as expected.

From what I gather, legacy TLS renegotiation would only kick in if the service for whatever reason didn't support RFC 5746. I'm skeptical that this would be the case for Key Vault, or indeed any Azure service, given that legacy SSL has a known vulnerability to MITM attacks dating back to 2009 (CVE-2009-3555).

The only situation I could imagine encountering something like this off the top of my head would be if accessing Key Vault via a proxy that only supports legacy SSL. I don't suppose that there could be anything like that in the mix with the setup you used to test this, @johnnyreilly?

[johnnyreilly]

Another possible workaround would be to set the agent property on the request in a custom pipeline policy (using the additionalPolicies option when configuring the client.)

Oh that's interesting - I'm not familiar with this approach. Are there docs?

[johnnyreilly]

The only situation I could imagine encountering something like this off the top of my head would be if accessing Key Vault via a proxy that only supports legacy SSL. I don't suppose that there could be anything like that in the mix with the setup you used to test this, @johnnyreilly?

That's a great question. This is done from a corporate network. And most traffic on that network is subject to SSL interception ( https://docs.netscaler.com/en-us/citrix-adc/13/forward-proxy/ssl-interception.html ) - probably with Palo Alto https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/decryption/configure-ssl-inbound-inspection

Do you think that's a likely cause?

[timovv]

I'd say that that's very likely to have something to do with it. With a proxy like that in effect, the client is performing the TLS handshake with the proxy instead of the Key Vault service, so all bets are off as to what happens. The proxy's CA certificate would also have to be trusted for this to work, which could cause problems, but if that was the issue I would have expected a more specific error.

In terms of attaching an additional policy as Jeff described above, there's no formal docs that I'm aware of -- this is quite an advanced use case. But it would look something like this (haven't tested; I think perCall is correct here but Jeff correct me if I'm wrong):

const client = new SecretClient(process.env.KEYVAULT_URI, new DefaultAzureCredential(), {
  additionalPolicies: [
    {
      policy: {
        name: "legacySSLRenegotiationPolicy",
        sendRequest(request, next) {
          request.agent = sslLegacyAgent;
          return next(request);
        },
      },
      position: "perCall",
    }
  ]
});

[xirzec]

Given that this is a private network issue rather than an SDK one, I'm not sure there's much we can do for this issue. I don't think we want to change the default behavior of the platform in a way that reduces security.

[timovv]

Agreed. To add to that, I would caution against using the workaround in a production environment given the security implications.

I'm going to close this issue, but feel free to drop any further questions here regardless :)

[johnnyreilly]

Thanks - using the workaround is a necessity for now, it's that or we have nothing that works. That said, I'll try and talk to our networking folk and see what they have to ssy

[johnnyreilly]

Just to follow up, we were able to confirm that the cause for this was Palo Alto performing SSL inspection and downgrading TLS along the way. We've got remedies in place on our side now so we don't need the workaround. Thanks for the discussion - this was useful input into our discussions. ❤️

[xirzec]

Just to follow up, we were able to confirm that the cause for this was Palo Alto performing SSL inspection and downgrading TLS along the way. We've got remedies in place on our side now so we don't need the workaround. Thanks for the discussion - this was useful input into our discussions. ❤️

Glad to hear your networking folks were able to help!

[fercod]

Just to follow up, we were able to confirm that the cause for this was Palo Alto performing SSL inspection and downgrading TLS along the way. We've got remedies in place on our side now so we don't need the workaround. Thanks for the discussion - this was useful input into our discussions. ❤️

Hi Jonny,
Could you please detail the workaround you did on the firewall? We also have paloalto and we are so lost, we don't know how to approach this. Thanks!

[johnnyreilly]

We have an allowlist of domains which are exempt from SSL interception. We added the Azure domains to that list