Loading of client certificates (PFX) crashes in Azure Functions sandbox

[maganvill]

Happens when app is deployed as Azure Function, not dev time.

Root cause is that CertUtil creates a new X509Certificate2 from client certificate byte array without setting the binding flags to "X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.Exportable" that is needed, that I learnt from previous experience in loading client certificates in Azure functions.

My idea is to let developer set the required flags in KubernetesClientConfiguration (default null of course). I will submit a PR.

Let's hope that you professionals will comment and lead me on the right way if wrong =)

Exception:

The SSL connection could not be established, see inner exception.
System.Net.Http.HttpRequestException : The SSL connection could not be established, see inner exception. ---> System.ComponentModel.Win32Exception : The credentials supplied to the package were not recognized
at System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface secModule,String package,CredentialUse intent,SCHANNEL_CRED scc)
at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(CredentialUse credUsage,SCHANNEL_CRED secureCredential)
at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(X509Certificate certificate,SslProtocols protocols,EncryptionPolicy policy,Boolean isServer)
at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint)
at System.Net.Security.SecureChannel.GenerateToken(Byte[] input,Int32 offset,Int32 count,Byte[]& output)
at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming,Int32 offset,Int32 count)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming,Int32 count,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer,Int32 count,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer,Int32 readBytes,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming,Int32 count,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer,Int32 count,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer,Int32 readBytes,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming,Int32 count,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer,Int32 count,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer,Int32 readBytes,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming,Int32 count,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer,Int32 count,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer,Int32 readBytes,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming,Int32 count,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer,Int32 count,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer,Int32 readBytes,AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Net.Security.SslState.ThrowIfExceptional()
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
at System.Net.Security.SslStream.<>c.b__47_1(IAsyncResult iar)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar,Func2 endFunction,Action1 endAction,Task1 promise,Boolean requiresSynchronization)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream,SslClientAuthenticationOptions sslOptions,CancellationToken cancellationToken)
End of inner exception
at async Microsoft.Rest.RetryDelegatingHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask,HttpRequestMessage request,CancellationTokenSource cts,Boolean disposeCts) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async k8s.Kubernetes.CreateNamespacedJobWithHttpMessagesAsync(V1Job body,String namespaceParameter,String dryRun,Nullable1 includeUninitialized,String pretty,Dictionary2 customHeaders,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async k8s.KubernetesExtensions.CreateNamespacedJobAsync(IKubernetes operations,V1Job body,String namespaceParameter,String dryRun,Nullable1 includeUninitialized,String pretty,CancellationToken cancellationToken)

[maganvill]

#237 solved my issue!

[pascaljr]

Could you guys provide a sample code on how to use this feature, please? I'm trying to use the Kubernetes API inside an Azure Function and I can't get it to work. Tnank you so much

[maganvill]

@pascaljr This is one way to do it

Unfortunately, Managed Identity is not supported dev time, like it is in Microsoft.Azure.Services.AppAuthentication. So for running locally, you must create a service principal for your client and give it the contributor role on your AKS resource.

Running in a Azure Function, Managed Identity is supported so use that SP instead and set the contributor role.

# Generate SP. Note: save the pwd
az ad sp create-for-rbac --name [xyz] --role contributor --scopes "/subscriptions/[subscriptionId]/resourcegroups/[your aks rg]/providers/Microsoft.ContainerService/managedClusters/[your aks resource name]"
# Give it access: 
az role assignment create --assignee [id from above result] --role contributor --scope "/subscriptions/
[subscriptionId]/resourcegroups/[your aks rg]/providers/Microsoft.ContainerService/managedClusters/[your aks resource name]"

You will need

• Nuget KubernetesClient 1.4.8-g9bbe42201f or greater
• Microsoft.Azure.Management.Fluent 1.19.0
• Microsoft.Azure.Management.ResourceManager.Fluent 1.19.0

// Get the Kube Config from ARM

// Debugging Locally
var credentials = SdkContext.AzureCredentialsFactory
   .FromServicePrincipal("[your sp app Id]", "[your sp pwd]", "[your tenantId]", AzureEnvironment.AzureGlobalCloud);

// On Azure Function
// var credentials = SdkContext.AzureCredentialsFactory
   .FromMSI(new MSILoginInformation(MSIResourceType.AppService), AzureEnvironment.AzureGlobalCloud);

var client = Azure
    .Configure()
    .Authenticate(credentials)
    .WithSubscription("[subscriptionId]");
var kubeConfigFileContent = client.KubernetesClusters
    .GetUserKubeConfigContentsAsync("[your aks rg]", "[your aks resource name]").Result;

// Get the k8s client
var stream = new MemoryStream(kubeConfigFileContent);
var config = KubernetesClientConfiguration.BuildConfigFromConfigFile(stream);
config.ClientCertificateKeyStoreFlags = 
   X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.Exportable;
var k8sClient = new Kubernetes(config);

[pascaljr]

@maganvill Thank you so much for taking the time and sharing the code with me. During the next few days I'm gonna try to use it.

[aravindyeduvaka]

Can we reopen this issue? I am actively hitting this on Azure Functions right now. However, it is not consistent. When it fails - it will continue to fail for a while then work at time. Makes me suspect that it might isolated to specific Azure Function Containers/VMs.

Here's the stacktrace:

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.ComponentModel.Win32Exception (0x8009030D): The credentials supplied to the package were not recognized at System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface secModule, String package, CredentialUse intent, SCHANNEL_CRED scc) at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(CredentialUse credUsage, SCHANNEL_CRED secureCredential) at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(X509Certificate certificate, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer) at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint) at System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count, Byte[]& output) at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest) --- End of stack trace from previous location where exception was thrown --- at System.Net.Security.SslStream.ThrowIfExceptional() at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult) at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result) at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult) at System.Net.Security.SslStream.<>c.b__65_1(IAsyncResult iar) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at k8s.WatcherDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at k8s.Kubernetes.CreateNamespacedSecretWithHttpMessagesAsync(V1Secret body, String namespaceParameter, String dryRun, String fieldManager, String pretty, Dictionary2 customHeaders, CancellationToken cancellationToken) at k8s.KubernetesExtensions.CreateNamespacedSecretAsync(IKubernetes operations, V1Secret body, String namespaceParameter, String dryRun, String fieldManager, String pretty, CancellationToken cancellationToken)