HttpGet Readiness Probe causes a deadlock

[PeterRockstars]

I am using https://github.com/dotnet-architecture/HealthChecks.
When an HttpGet Readiness Probe is configured, either Application Insights Kubernetes fails to start, or the pod never gets to Ready.

Steps to reproduce:

add this to deployment.yaml:

readinessProbe:
  httpGet:
    path: /hc
    port: 80
    scheme: HTTP
  initialDelaySeconds: 130 #(10 seconds more than the default timeout)
  periodSeconds: 10
  failureThreshold: 10
  timeoutSeconds: 10

This will cause the pod to be ready after 130 seconds, but the logs show:

fail: Microsoft.ApplicationInsights.Kubernetes.K8sEnvironment[0]
Kubernetes info is not available within given time of 120000 ms.
fail: K8sEnvInitializer[0]
Application Insights Kubernetes failed to start.

When the probe is defined like this:

readinessProbe:
  httpGet:
    path: /hc
    port: 80
    scheme: HTTP
  initialDelaySeconds: 10 #(start probing after 10 seconds))
  periodSeconds: 10
  failureThreshold: 10
  timeoutSeconds: 10

the pod never reaches ready state and gives an error:

Readiness probe failed: Get http://xx.xxx.x.122:80/hc: dial tcp xx.xxx.x.122:80: getsockopt: connection refused

The logs show that the application never initialises further than this message:

trce: Microsoft.ApplicationInsights.Kubernetes.KubeHttpClientSettingsProvider[0]
System.Security.Cryptography.X509Certificates.X509Chain
trce: Microsoft.ApplicationInsights.Kubernetes.KubeHttpClientSettingsProvider[0]
RemoteCertificateChainErrors
trce: Microsoft.ApplicationInsights.Kubernetes.KubeHttpClientSettingsProvider[0]
ServerCert:
[Subject]
CN=apiserver

[Issuer]
CN=acsk8sca

[Serial Number]
XXXXXXXXXXXXX

[Not Before]
10/26/2017 00:00:00

[Not After]
10/26/2019 00:00:00

[Thumbprint]
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

dbug: Microsoft.ApplicationInsights.Kubernetes.KubeHttpClientSettingsProvider[0]
Issuer validation passed: CN=acsk8sca
dbug: Microsoft.ApplicationInsights.Kubernetes.KubeHttpClientSettingsProvider[0]
Server certificate validate date verification passed.
dbug: Microsoft.ApplicationInsights.Kubernetes.KubeHttpClientSettingsProvider[0]
Server certification custom validation successed.

A possible solution might be to wrap the initialisation in a Task.Run(), as is done for retrieving the version info in KubernetesModule.EnableKubernetes

[xiaomi7732]

@PeterRockstars,
Thanks for the details. We will look into this post 1.0.0-beta2.

[xiaomi7732]

This issue is fixed and will be released in the next release.

[PeterRockstars]

Fantastic! Do you know yet when the next release will be?

[xiaomi7732]

@PeterRockstars,
Thanks for the ask. The next release is due by 1/12/2018. As always, you will find the info here: https://github.com/Microsoft/ApplicationInsights-Kubernetes/milestones

[PeterRockstars]

So polite! Thanks 👍