[QUERY]Initial request from function app to retrieve key vault secret results in 401 #38039
Comments
github-actions
bot
added
Client
|
Hi @jason-goodlife. Thank you for reaching out and we regret that you're experiencing difficulties. In order to assist, we'll need some additional context to better understand the scenario. Can you share the full error message and stack trace? It would also be helpful to understand where you're calling the KeyVault client snippet that you shared. |
jsquire
added
needs-author-feedback
|
Hi @jason-goodlife. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue. |
|
I see that, we are facing the same issue, and also in case of long running application and lets say we want to fetch a secret after 3 days, the cancellationtoken timeout includes the authentication time also, so after some days the requests will not complete as they completed at startup as the initial requests had more timeout and the requests we want to have in middle of the application run has lower timeout. To tackle this one, i created a method where it fetches a dummy secret synchronously within try catch block, just before the getSecret call in sometime, like after every 1 hour is elapsed. but with a higher timeout like 5 sec. |
|
Currently the application code throws no exceptions, therefore, no stack trace is provided. I will attempt to wrap the code in a Try/Catch to see if I can gather any more details. The 401 exceptions only appear on the Dependencies tab in the Failure blade in Application Insights. The Operations nor the Exceptions tabs on the Failures blade lists the exceptions. The key vault secrets are fetched from within the different methods of the function app. These are the 2 entries from the traces log in our Application Insights for the function app. These 2 log entries are from a single request in our application to our key vault. First request results in a 401 timestamp [UTC] 2023-08-08T13:43:08.9485659Z message Request [416753c9-96c3-460b-997f-87f04550a014] Second request results in a 200 with the secret value returned, as expected. timestamp [UTC] 2023-08-08T13:43:10.0509086Z message Request [416753c9-96c3-460b-997f-87f04550a014] As you can see, the first request has no authorization. The second request does have authorization (redacted). The request retry is not attempted by our application code so I assume this is part of the GetSecretAsync method in the Azure.Security.KeyVault.Secrets nuget package. The time between the failed and successful requests in this case was seconds. We are experiencing the same thing as @vipinjn24 mentioned above with regards to when the key vault request fails affecting "start up" time and responding to the initial request even though the key vault secrets are requested by many of the methods in the function app and not by any startup code in the function app. |
github-actions
bot
added
needs-team-attention
|
I added a Try/Catch around the call to GetSecretAsync and from the application code POV, no exception or error is thrown and ultimately, after a 401 then retry, the key vault secret is returned after many seconds to re-authenticate the initial request. For clarity, here is the code fetching the key vault secret, using Azure.Identity;
using Azure.Security.KeyVault.Secrets;
using Microsoft.Extensions.Logging;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
namespace FunctionAppOrchestration;
public static class KeyVaultHelpers {
private static readonly Dictionary<string, string> _secrets = new Dictionary<string, string>();
private static readonly DefaultAzureCredentialOptions _credentialOptions = new DefaultAzureCredentialOptions();
private static readonly DefaultAzureCredential _azureCredentials = new DefaultAzureCredential(_credentialOptions);
public static async Task<string> GetSecret(string secretUrl, ILogger logger) {
try {
logger.LogInformation("***** getting secret");
if (string.IsNullOrWhiteSpace(secretUrl)) return string.Empty;
if (!secretUrl.Contains("vault.azure")) return secretUrl;
if (!_secrets.ContainsKey(secretUrl)) {
logger.LogInformation("***** secret not found in list");
string keyVaultUrl = secretUrl[..secretUrl.LastIndexOf("/secrets")];
logger.LogInformation("***** creating secret client");
SecretClient client = new SecretClient(new Uri(keyVaultUrl), _azureCredentials);
logger.LogInformation("***** fetch secret from key vault");
KeyVaultSecret secret = await client.GetSecretAsync(new Uri(secretUrl).Segments.Last());
logger.LogInformation("***** adding secret value to list");
_secrets.Add(secretUrl, secret.Value);
}
return _secrets[secretUrl];
}
catch (Exception ex) {
logger.LogInformation($"Get secret threw an exception - {ex.Message}");
throw;
}
}
}
And here are the log entries created by the code above in the traces table in Application Insights, |
Thank you for the clarification and apologies for the confusion. I misunderstood and thought you were seeing the 401 trigger an application exception, which would indicate that it persisted until all implicit retries had been exhausted. That would be a sign that something is wrong. A single 401 on initial request is expected and can safely be ignored. This will occur regardless of which credential type is used - it occurs due to Key Vault's use of challenge-based auth, which is discussed (somewhat briefly) here. If you're looking to speed up the process and you're not using managed identity, disabling it as discussed in the issue that you linked may help, as would using a more narrowly scoped credential type such as I'm going to mark this as addressed, but please feel free to unresolve if you'd like to continue the discussion. |
jsquire
added
issue-addressed
github-actions
bot
removed
the
needs-team-attention
|
Hi @jason-goodlife. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation. |
|
Thanks for the clarification @jsquire. Much appreciated. As a follow up and further education.... What mechanism is in place to swallow the exception when I use a key vault reference in the application configuration settings? When I use a key vault reference no exceptions are thrown in AI. Clearly the exception is being swallowed or simply not reported to AI when using a key vault reference? |
Unfortunately, I don't have the answer to that, as it would be a behavior of the Functions runtime and not the extensions packages. Your assessment seems reasonable to me. I would guess that there's filtering logic. |
|
Hi @jason-goodlife, since you haven’t asked that we |
Library name and version
Azure.Security.KeyVault.Secrets 4.5.0
Query/Question
We have an Azure Function App recently upgraded from v1 to v4. The function app is throwing 401's on the initial request to retrieve a key vault secret. We cannot confirm if this has always been happening or if this is a result from the upgrade to v4.
The function app is now v4 in C# .NET 6 using the Azure.Security.KeyVault.Secrets nuget package. This is the full list of all nuget packages for the function app.
Here is the simplified code used to fetch the key vault secret,
All methods in this function app are static.
The code clearly works, eventually, after the first request throws a 401 because there is no authentication token provided. The request is then re-authenticated auto-magically and retried. With the valid token in place, the request is then successful and the key vault secret is returned.
I found this GitHub issue and tried to implement a startup class to configure the default Azure credentials,
#29471
But when using the startup class, the function app threw this startup error,
https://learn.microsoft.com/en-us/azure/azure-functions/errors-diagnostics/diagnostic-events/azfd0005
I changed tactics and was able to prevent the 401's from happening by using a key vault reference in our function app configuration.
https://learn.microsoft.com/en-us/azure/app-service/app-service-key-vault-references?tabs=azure-cli
My question is if this is the intended behaviour when fetching secrets from our key vault using DefaultAzureCredential? If not, is the only method to prevent the 401 exceptions is to use key vault references in our configuration? My concern is that we are receiving these 401 exceptions on many of our Azure app services, not just this function app and they are all using similar implementations to use SecretClient and KeyVaultSecret using DefaultAzureCredential.
Environment
C#
.NET 6
Azure Function App v4
Key Vault
The text was updated successfully, but these errors were encountered: