Istio readiness Probe rewrite should respect the user specified timeout duration #18242
Comments
howardjohn
added
area/security
and removed
area/networking
area/extensions and telemetry
labels
|
Do you mind do
|
|
@incfly example-app logs Istio proxy logs Environment variables istio-proxy: PS: When I deploy app using |
This is what you pasted from the env var section, which is different from
which one is the actual accurate env var of the istio-proxy container? |
|
Also what's your app health check endpoint time out/latency duration? I vaguely remembered we have a hard-coded in pilot agent log, if your app is longer than that, we may pass the timeout. incfly@b412d59#diff-c1bdb159b354a382bbfe6a0bb13863b8R156 ours is 10 seconds, might be less than kubelet's implementation... |
|
@incfly When I add My app health check timeout is 10 seconds. |
|
okay, i see the kubelet code has a time out duration specified, we didn't respect that, rather using a hardcoded 10 seconds. https://sourcegraph.com/github.com/kubernetes/kubernetes@a381f7cb3efa64395661c73961c07b02594acf64/-/blob/pkg/kubelet/prober/prober.go#L172 In your case, you explicitly set the time out to be 10 seconds, and we're on the boundary line for this, so we might time out too early. I think we can have a fix to propogate the timouet to istio pilot-agent to respect the timeout. BTW, I don't quite understand the What I'm expecting with sidecar injected, without sidecar injected, no env var. ( I just want to ensure whether your health check works before for a control comparision) |
incfly
changed the title
|
I have auto-inject enabled but it should overwrite global setting over deployment values. Please take a look at below logs (sidecar inject enabled): As you can see, |
probers point to a new endpoint at port 15020, this is expected. As the name indicates, "prober rewrites" means the actual podspec container prober configuration is configured to different path and port. The buggy part is that we miss the delay & time out section... which needs to be implemented in sidecar injector, and respect the original timeout as well.. |
|
I do agree we miss the timeout part. Which should be carried over to the pilot agent -> app. But I can't tell anything else is wrong. also given your apps timeout is 10s, which equals to our hardcoded time out in our pilot agent -> app, so I'm not sure fixing that will address your issue.. Actually I think because we use I confirmed in my own deployment with a readinessProbe.initialDelaySeconds = 50s, and that's kept untouched after rewrite. Are you sure you set original prober's delay seconds to 700 seconds and then you see the injected one, app probers become 200s?
This error is complaining about the sidecar its own health check is failing, which should not be related to app probe rewrite... I dont think if such 200/700 initial delay of the apps have anything to do with sidecar's health check though. |
|
@incfly 700s to 200s I changed it. So, that's not the issue (sorry for the confusion). But, When I set it to 200s it shouldn't check for readiness before that. The app should be in ready state after 200 seconds, in this case, it happens immediately which is incorrect I guess. Even if it's showing timeout in the Initialdelay is 200 sec, container is not ready to serve traffic but still, the state is Running. |
|
healthz/ready is for the sidecar, istio-proxy container, health check. It's
there regardless you enable the rewrite or not. It's just the path
collision introducing some confusion.
…On Mon, Nov 4, 2019 at 6:38 AM rnkhouse ***@***.***> wrote:
@incfly <https://github.com/incfly> 700s to 200s I changed it. So, that's
not the issue (sorry for the confusion). But, When I set it to 200s it
shouldn't check for readiness before that.
This happened in 60 sec which is incorrect I guess.
Warning Unhealthy 63s (x2 over 64s) kubelet, aks-agentpool-17141372-vmss000003 Readiness probe failed: HTTP probe failed with statuscode: 503
Warning Unhealthy 60s kubelet, aks-agentpool-17141372-vmss000003 Readiness probe failed: Get http://10.0.11.55:15020/healthz/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Even if it's showing timeout in the pod description the app is in running
state.
Status: Running
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#18242?email_source=notifications&email_token=AAMVXB4Z46Y5DEH4SGXCAF3QSAXYBA5CNFSM4JEIKLS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC7OZBY#issuecomment-549383303>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMVXB2DNRX7LL3A4YE2WXDQSAXYBANCNFSM4JEIKLSQ>
.
|
|
Here are some pointers of where the changes can be made to pass the timeout
Hope this helps @johnma14 |
incfly
added
lifecycle/staleproof
istio-policy-bot
added
lifecycle/needs-triage
and removed
lifecycle/needs-triage
labels
|
The same problem occurred in my environment, but it was due to a "double Istio injection".
No problem if only one of them. https://istio.io/docs/ops/configuration/mesh/app-health-check/#enable-globally-via-install-option fine too. Solution Example: $ kubectl get cm istio-sidecar-injector -n istio-system -o yaml | sed -e 's/"rewriteAppHTTPProbe":false/"rewriteAppHTTPProbe":true/' | kubectl apply -f -
configmap/istio-sidecar-injector configured
$ kubectl label namespace default istio-injection=enabled --overwrite
namespace/default labeled
$ kubectl apply -f samples/health-check/liveness-http-same-port.yaml
service/liveness-http created
deployment.apps/liveness-http createdOr $ kubectl get cm istio-sidecar-injector -n istio-system -o yaml | sed -e 's/"rewriteAppHTTPProbe":false/"rewriteAppHTTPProbe":true/' | kubectl apply -f -
configmap/istio-sidecar-injector configured
$ kubectl label namespace default istio-injection=disabled --overwrite
namespace/default labeled
$ kubectl apply -f <(istioctl kube-inject -f samples/health-check/liveness-http-same-port.yaml)
service/liveness-http created
deployment.apps/liveness-http createdMy version: $ kubectl version | awk '{print $5}'
GitVersion:"v1.17.0",
GitVersion:"v1.15.7",
$ istioctl version
client version: 1.4.0
control plane version: 1.4.0
data plane version: 1.4.0 (3 proxies)
$ sudo minikube version
minikube version: v1.4.0
commit: 7969c25a98a018b94ea87d949350f3271e9d64b6 |
…ied timeout The httpClient created in the pilot agent code currently uses the default timeout of 10s. Updated the code to use the timeout specified by the user instead. Fixes: istio#18242
…tio#18242) * support timeout retention on livez and readyz probe rewrite (fixes istio#18242) * fix incorrect error formatting * fix tests for pilot-agent * use internal struct for prober configuration
Bug description
I followed the guide from this link (https://istio.io/docs/ops/app-health-check/) and using Enable via Helm Option Globally method to use readiness probe in my application. sidecarInjectorWebhook.rewriteAppHTTPProbe=true has been set in global config.
Deployment
Error
Expected behavior
It should check readiness probe and route traffic if it's ready. Also, it should point to the application's
/healthzpath for readiness check as described in deployment settings.Version (include the output of
istioctl version --remoteandkubectl version)Istio version: 1.2.6
Kubernetes version: 1.14.5
How was Istio installed?
Helm template
Environment where bug was observed (cloud vendor, OS, etc)
AKS
The text was updated successfully, but these errors were encountered: