Azure Function: The client certificate credentials were not recognized

[dring1]

Running an Azure Function locally will result in the kubernetes client successfully communicating with the agents a given k8s cluster on azure.

When running the function on the azure functions environment I receive a client certificate credentials were not recognized error, despite the creds being identical on azure and local.

Investigative information

Please provide the following:

• Timestamp: 2018-01-09T15:34:40.618
• Function App version (1.0 or 2.0-beta): 2.0-beta
• Function App name:
• Function name(s) (as appropriate): NodeDestroyer
• Invocation ID: a2129799-8b25-4e1b-9731-56e4a13b6b10
• Region: CentralUSPlan

Repro steps

Provide the steps required to reproduce the problem:

Linking to Kubernetes CSharp client repo issue : kubernetes-client/csharp#79

1. Run azure function locally
2. Publish to azure and start function

Expected behavior

Provide a description of the expected behavior.

Azure function executes identically to local development when published and started on azure.

Actual behavior

Provide a description of the actual behavior observed.

Running an Azure Function locally will result in the kubernetes client successfully communicating with the agents a given k8s cluster on azure.

When running the function on the azure functions environment I receiving the follow stacktrace

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: NodeDestroyer ---> System.Net.Http.HttpRequestException : An error occurred while sending the request. ---> System.Net.Http.WinHttpException : The client certificate credentials were not recognized
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.WinHttpHandler.StartRequest(WinHttpRequestState state) 
   End of inner exception
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async k8s.WatcherDelegatingHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Rest.RetryDelegatingHandler.<>c__DisplayClass11_0.<SendAsync>b__1(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Rest.RetryDelegatingHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask,HttpRequestMessage request,CancellationTokenSource cts,Boolean disposeCts)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async k8s.Kubernetes.ListNamespacedServiceWithHttpMessagesAsync(String namespaceParameter,String continueParameter,String fieldSelector,Nullable`1 includeUninitialized,String labelSelector,Nullable`1 limit,String resourceVersion,Nullable`1 timeoutSeconds,Nullable`1 watch,String pretty,Dictionary`2 customHeaders,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async k8s.KubernetesExtensions.ListNamespacedServiceAsync(IKubernetes operations,String namespaceParameter,String continueParameter,String fieldSelector,Nullable`1 includeUninitialized,String labelSelector,Nullable`1 limit,String resourceVersion,Nullable`1 timeoutSeconds,Nullable`1 watch,String pretty,CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at k8s.KubernetesExtensions.ListNamespacedService(IKubernetes operations,String namespaceParameter,String continueParameter,String fieldSelector,Nullable`1 includeUninitialized,String labelSelector,Nullable`1 limit,String resourceVersion,Nullable`1 timeoutSeconds,Nullable`1 watch,String pretty)
   at InfrastructureManager.NodeDestroyer.TakeAvailableService(String id,TraceWriter log,ExecutionContext context) at C:\GitHub\WebContent\InfrastructureManager\InfrastructureManager\InfrastructureManager.cs : 129
   at InfrastructureManager.NodeDestroyer.CreateAvailableNodes(TraceWriter log,ExecutionContext context) at C:\GitHub\WebContent\InfrastructureManager\InfrastructureManager\InfrastructureManager.cs : 57
   at InfrastructureManager.NodeDestroyer.Run(TimerInfo myTimer,TraceWriter log,ExecutionContext context) at C:\GitHub\WebContent\InfrastructureManager\InfrastructureManager\InfrastructureManager.cs : 38
   at lambda_method(Closure ,NodeDestroyer ,Object[] )
   at Microsoft.Azure.WebJobs.Host.Executors.VoidMethodInvoker`2.InvokeAsync(TReflected instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\VoidMethodInvoker.cs : 20
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 63
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 583
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 534
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 477
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 264 
   End of inner exception

The only difference from azure function local dev to production is how the environment variables container the certs are ingested.

I have hardcoded the Client credentials to what would be available in azure, and the error+stacktrace persists when ran on azure, but successful when ran locally.

Known workarounds

Provide a description of any known workarounds.

Related information

Provide any related information

• Programming language used
  C# Class Library Azure function
• Links to source
• Bindings used

Source

		[FunctionName("NodeDestroyer")]
		public static void Run([TimerTrigger("*/30 * * * * *")]TimerInfo myTimer, TraceWriter log, ExecutionContext context)
		{
			// explicit + simplified 
			var endpoint = System.Environment.GetEnvironmentVariable("NodeManagerEndpoint");
			nodesvc = new NodeManagerAPIService(endpoint);
			var kubeconfig = new KubernetesClientConfiguration();
			var clientCertData = Environment.GetEnvironmentVariable("ClientCertificateData");
			kubeconfig.ClientCertificateData = clientCertData;
			var clientCertKeyData = Environment.GetEnvironmentVariable("ClientCertificateKeyData");
			kubeconfig.ClientCertificateKeyData = clientCertKeyData;
			var sslCaCert = Environment.GetEnvironmentVariable("SslCaCert");
			X509Certificate2 cert = new X509Certificate2(Convert.FromBase64String(sslCaCert));
			kubeconfig.SslCaCert = cert;
			kubeconfig.Host = Environment.GetEnvironmentVariable("Host");

			kubectl = new Kubernetes(kubeconfig);


			EliminateDestroyableNodes(log, context);
			EliminatedDestroyedNodes(log);
			CreateAvailableNodes(log, context);
			log.Info($"NodeDestroyer Timer trigger function executed at: {DateTime.Now}");
		}

[dring1]

updated stacktrace with additional info

[dring1]

This was root caused as an azure related issue;
Must use the X509 Store to consume an ssl cert and use it against an Kubernetes cluster.

using System;

using System.Security.Cryptography.X509Certificates;namespace UseCertificateInAzureWebsiteApp

{

  class Program

  {

    static void Main(string[] args)

    {

      X509Store certStore = new X509Store(StoreName.My, StoreLocation.CurrentUser);

      certStore.Open(OpenFlags.ReadOnly);

      X509Certificate2Collection certCollection = certStore.Certificates.Find(

                                 X509FindType.FindByThumbprint,

                                 // Replace below with your cert's thumbprint

                                 “E661583E8FABEF4C0BEF694CBC41C28FB81CD870”,

                                 false);

      // Get the first cert with the thumbprint

      if (certCollection.Count > 0)

      {

       X509Certificate2 cert = certCollection[0];

        // Use certificate

        Console.WriteLine(cert.FriendlyName);

      }

      certStore.Close();

    }

  }

}