Random 503 errors with liveness probes

[day0ops]

Istio: 0.7.1
K8: 1.9.7
mTLS: Disabled

I see a number of random 503 errors causing my pods to crash (randomly)

[2018-05-26T04:17:41.562Z] "GET /management/health HTTP/1.1" 200 - 0 103 268 267 "-" "kube-probe/1.9+" "c5503c80-d6aa-965d-b617-c2d4ea31c2fa" "10.4.0.9:9079" "127.0.0.1:9079"
[2018-05-26T04:44:56.408Z] "GET /management/health HTTP/1.1" 503 UC 0 57 358 - "-" "kube-probe/1.9+" "c7706a1d-c07e-9c19-8f8d-091f11f6dd8a" "10.4.0.9:9079" "127.0.0.1:9079"
[2018-05-26T04:48:23.669Z] "GET /management/health HTTP/1.1" 200 - 0 103 504 264 "-" "kube-probe/1.9+" "98d718d1-1821-92ce-a7a0-fbdf52503c86" "10.4.0.9:9079" "127.0.0.1:9079"
[2018-05-26T05:03:11.405Z] "GET /management/health HTTP/1.1" 200 - 0 103 545 13 "-" "kube-probe/1.9+" "1d9fb6b6-8503-9e77-845c-efbe24ff6a2e" "10.4.0.9:9079" "127.0.0.1:9079"
[2018-05-26T05:45:26.440Z] "GET /management/health HTTP/1.1" 503 UC 0 57 305 - "-" "kube-probe/1.9+" "6fffcfc8-38c0-9fb7-96d7-e6a90da91db9" "10.4.0.9:9079" "127.0.0.1:9079"
[2018-05-26T06:35:53.616Z] "GET /management/health HTTP/1.1" 200 - 0 103 127 73 "-" "kube-probe/1.9+" "53e0570c-a0cd-90a0-9328-550300eefd08" "10.4.0.9:9079" "127.0.0.1:9079"
[2018-05-26T06:38:26.395Z] "GET /management/health HTTP/1.1" 200 - 0 103 85 28 "-" "kube-probe/1.9+" "faf36503-05ab-9b3b-a2c6-74a899d65178" "10.4.0.9:9079" "127.0.0.1:9079"

Im not sure what causes this. Has anyone successfully got the liveness probes to work ?

I have a number of Spring Boot microservices that I have configured with separate port to run the healthchecks on based on recommendations I have seen.

For instance:

readinessProbe:
  httpGet:
    scheme: HTTP
    path: /management/health
    port: 9079
  initialDelaySeconds: 300
  periodSeconds: 15
  timeoutSeconds: 15
  successThreshold: 1
  failureThreshold: 10
livenessProbe:
  httpGet:
    scheme: HTTP
    path: /management/health
    port: 9079
  initialDelaySeconds: 300
  periodSeconds: 15
  timeoutSeconds: 15
  successThreshold: 1
  failureThreshold: 3

And in the service I have,

  ....
  annotations:
    auth.istio.io/9079: NONE

[sakshigoel12]

@nixgadget please could you upgrade to 0.8 and suggest if you still see this issue

[jasminejaksic]

@nixgadget are you encountering this issue with 0.8?

[costinm]

With 0.8 it should not be a problem: 0.8 does not intercept any port not declared as container port, and since you use different port the request doesn't use the sidecar.

This may change - #6259 - closing this bug since it's not reproductible or actionable at this point, we'll need to make sure 6259 PR doesn't add this back.