[BUG] Azure Identity credentials don't obey proxy when acquiring login tokens

[paulmedynski-microsoft]

Library name and version

Azure.Security.KeyVault.Secrets 4.6.0

Describe the bug

Our hosts require HTTPS proxy for all egress. We can configure the SecretClientOptions.Transport with a suitable proxy (see attached sample program), and requests to the Azure KeyVault obey this proxy. However, requests from within SecretClient to obtain login tokens do not obey the proxy, and thus they fail to complete. If a proxy is set via SecretClientOptions.Transport, it should be used for all requests made by SecretClient or any libs that it uses.

Expected behavior

A proxy set viat SecretClientOptions should be used to all requests made by, or on behalf of, SecretClient.

Actual behavior

Requests made by SecretClient to acquire login tokens do not use the configured proxy.

The only way for login tokens to be acquired successfully is to change the HttpClient.DefaultProxy, which globally affects all HttpClient instances - obviously not a suitable solution.

Reproduction Steps

Use this project to reproduce the issue:

https://github.com/paulmedynski/azure-key-vault-proxy

Run with --help for options related to Azure Key Vault and whether or not a proxy is used.

All egress allowed, no proxy specified:

$ dotnet run -- -u https://<keyvault>/ -t <tenantId> -a <appId> -c <cert.pem> -k my-secret-name

11:54:30.338 info: KeyVaultProxy[0] Creating SecretClient for vault URL=https://<keyvault>/ tenantId=<tenantId> appId=<appId>...
11:54:30.351 info: KeyVaultProxy[0] Using auth type=cert (cert.pem)
11:54:30.353 info: KeyVaultProxy[0] Getting secret for key=my-secret-name...
11:54:31.606 info: KeyVaultProxy[0] Secret for key=my-secret-name is: id=https://<keyvault>/secrets/my-secret-name/945c0e7946314f90bb3d5e6f74e1d120 name=my-secret-name value=shhh_secret

No egress allowed, no proxy specified:

Configure an HTTPS proxy on localhost:8888, for example tinyproxy.

Configure iptables to reject all outbound traffic except from the proxy:

# iptables -F OUTPUT
# iptables -A OUTPUT -j ACCEPT -m owner --uid-owner tinyproxy
# iptables -A OUTPUT -j ACCEPT -o lo
# iptables -A OUTPUT -j REJECT

$ dotnet run -- -u https://<keyvault>/ -t <tenantId> -a <appId> -c <cert.pem> -k my-secret-name

14:34:13.245 info: KeyVaultProxy[0] Creating SecretClient for vault URL=https://<keyvault>/ tenantId=<tenantId> appId=<appId>...
14:34:13.258 info: KeyVaultProxy[0] Using auth type=cert (cert.pem)
14:34:13.260 info: KeyVaultProxy[0] Getting secret for key=my-secret-name...
14:34:19.074 fail: KeyVaultProxy[0] Failed to fetch secret for key=my-secret-name: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (Connection refused (<keyvault>:443)) (Connection refused (<keyvault>:443)) (Connection refused (<keyvault>:443)) (Connection refused (<keyvault>:443))

Note that the initial connection to the Key Vault is refused, as expected.

No egress allowed, SecretClient proxy specified:

$ dotnet run -- -u https://<keyvault>/ -t <tenantId> -a <appId> -c <cert.pem> -k my-secret-name -p localhost:8888

14:36:14.837 info: KeyVaultProxy[0] Creating SecretClient for vault URL=https://<keyvault>/ tenantId=<tenantId> appId=<appId>...
14:36:14.848 info: KeyVaultProxy[0] Using auth type=cert (cert.pem)
14:36:14.848 info: KeyVaultProxy[0] Using proxy localhost:8888
14:36:14.850 info: KeyVaultProxy[0] Getting secret for key=my-secret-name...
14:36:21.633 fail: KeyVaultProxy[0] Failed to fetch secret for key=my-secret-name: ClientCertificateCredential authentication failed: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (Network is unreachable (login.microsoftonline.com:443)) (Network is unreachable (login.microsoftonline.com:443)) (Network is unreachable (login.microsoftonline.com:443)) (Network is unreachable (login.microsoftonline.com:443)) See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/clientcertificatecredential/troubleshoot

Now we see that the login token requests are failing, because they aren't using the proxy! This can be corroborated by inspecting the proxy logs and noting that only the initial Key Vault was tunneled through the proxy. The login token requests did not pass through the proxy.

No egress allowed, HttpClient.DefaultProxy specified:

$ dotnet run -- -u https://<keyvault>/ -t <tenantId> -a <appId> -c <cert.pem> -k my-secret-name -p localhost:8888 --global-proxy

14:37:50.710 info: KeyVaultProxy[0] Creating SecretClient for vault URL=https://<keyvault>/ tenantId=<tenantId> appId=<appId>...
14:37:50.720 info: KeyVaultProxy[0] Using auth type=cert (cert.pem)
14:37:50.721 info: KeyVaultProxy[0] Using proxy localhost:8888  (global)
14:37:50.722 info: KeyVaultProxy[0] Getting secret for key=my-secret-name...
14:37:52.433 info: KeyVaultProxy[0] Secret for key=my-secret-name is: id=https://<keyvault>/secrets/my-secret-name/945c0e7946314f90bb3d5e6f74e1d120 name=my-secret-name value=shhh_secret

Everything works when the global HttpClient.DefaultProxy is specified. This isn't a suitable solution though, since other HTTP connections within the app may rely on a default proxy specified by environment variables for example.

Environment

$ dotnet --info
.NET SDK:
Version: 8.0.202
Commit: 25674bb2f4
Workload version: 8.0.200-manifests.4e94be9c

Runtime Environment:
OS Name: debian
OS Version: 12
OS Platform: Linux
RID: linux-x64
Base Path: /usr/share/dotnet/sdk/8.0.202/

.NET workloads installed:
There are no installed workloads to display.

Host:
Version: 8.0.3
Architecture: x64
Commit: 9f4b1f5d66

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jsquire]

@christothes: Would you please offer your thoughts?

[christothes]

Hi @paulmedynski-microsoft
Could you try passing options to your credential with the Transport set the same way as in your SecretClient?

Some additional context:
TokenCredential implementations in Azure.Identity are essentially their own clients and the TokenCredentialOptions implementation accepted by each credential uses its own transport and pipeline. Since TokenCredentialOptions inherits from ClientOptions, the same transport options are settable there

azure-sdk-for-net/sdk/identity/Azure.Identity/src/Credentials/TokenCredentialOptions.cs

Line 13 in 5c7c954

public class TokenCredentialOptions : ClientOptions

[github-actions]

Hi @paulmedynski-microsoft. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[paulmedynski-microsoft]

Thanks @christothes - that worked perfectly! I have updated my sample app with the fix.

It might help future devs to somehow mention the additional context you provided above somewhere here:

https://learn.microsoft.com/en-us/dotnet/api/azure.security.keyvault.secrets?view=azure-dotnet

Or perhaps under a new Credentials section.

Thank you for the prompt reply and solution!