Repository navigation
envoy drain duration should adapt to pod spec #34855
Description
Activity
We use
--drain-strategy immediatewhich should somewhat help "envoy's drain is passive" if you weren't aware - doesn't work in all cases though.Overall I agree we should do this. A few concerns I think we will need to work out:
- what we should do if users set
terminationDrainDuration(or just have the old default of 5s)? - Will we hit any cases where we get stuck? For example, I have seen some users set terminationGracefulPeriodSeconds to extreme values (>10 minutes) - if there are cases where their pod would have shut down and now it doesn't we may run into issues
The benefits largely outweigh the issues, but I we should see what we can do to address those
- what we should do if users set
@howardjohn It's passive because Envoy does not signal clients pro-actively. On a long-living connection with few requests, clients simply don't receive any notification. For very long grace periods, we have to implement polling for open connections so that if app exits early, proxy does, too. If we do pursue this, I'd imagine flags (1)-(3) are no longer useful. 5s is a dangerously low value, more so for apps with very longs shutdowns.
One complication is that health checks (and some other endpoints) go through Envoy and those should not count as application connections during drain.
Tagging @hzxuzhonghu.
One complication is that health checks (and some other endpoints) go through Envoy and those should not count as application connections during drain.
This is exactly the same reason why we can not completely depend on Envoy to provide draining connection count.
I just ran into this issue; one of my services has a high maximum but low average request time, so it gracefully shuts itself down, but once it's actually exited the sidecar persists and waits around even though there's no active connections.
Hopefully I see this in a release soon, because I'd really like to test it and have even faster resource freeing across our clusters.
@kevin-lindsay-1 #35059 has this implementation. You can set EXIT_ON_ZERO_ACTIVE_CONNECTIONS on Agent side and try it. Please let us know how it works
@ramaraochavali I use helm to install istio, and can change the image tags. Are there image builds that I could try out?
https://storage.googleapis.com/istio-build/dev/latest - Please try with this
@ramaraochavali I use dockerhub for my images; can you tell me what I'd need to change on the helm chart for
istio-discoveryto test that image, because I'm getting an error when trying to switch stuff to that hub/tag.Sorry. I am not aware of that. May be @howardjohn knows about it
- --set global.hub, --set global.tag…On Mon, Oct 11, 2021 at 10:30 PM Rama Chavali ***@***.***> wrote: Sorry. I am not aware of that. May be @howardjohn <https://github.com/howardjohn> knows about it — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34855 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXLMRZ6OZO44EQKOBLLUGPBXXANCNFSM5CX26JXQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
@howardjohn sorry, I understand the
valuesto set, but I don't know what the actualhubandtagvalues would be, as this appears to be a google hub and tag, and the attempts I made all gave me errors.Not experienced with using google storage as a hub, to the point where I'm not even 100% that it's compatible as an implementation of a docker hub.
- gcr.io/istio-testing tag=latest https://github.com/istio/istio/wiki/Dev%20Builds…On Tue, Oct 12, 2021 at 3:57 PM Kevin Lindsay ***@***.***> wrote: @howardjohn <https://github.com/howardjohn> sorry, I understand the values to set, but I don't know what the actual hub and tag values would be, as this appears to be a google hub and tag, and the attempts I made all gave me errors. Not experienced with using google storage as a hub, to the point where I'm not even 100% that it's compatible as an implementation of a docker hub. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34855 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXKBF46II7UPDNQ4ARLUGS4LXANCNFSM5CX26JXQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
44 remaining items
Sorry I have been busy with other stuff. PR for backport #39286 to 1.12
🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2022-06-05. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.
Created by the issue and PR lifecycle manager.
Hi @ramaraochavali ,
While testing the settingEXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true',I deliberately kept an active connection and killed the pod usingkubectl delete,the envoy logs says one active connection clearly but after theterminationGracefulPeriodtheistio-proxy + appis killed with active connection, Should there be a doc to say that user needs to adjust theterminationGracefulPeriodif the app takes more time than default30s.Sure. We can update the docs
EXIT_ON_ZERO_ACTIVE_CONNECTIONS isn't mentioned in latest docs - has the default changed to true or it still exists in current istio as an option? and maybe related is there an uptodate doc on how to ensure the proxy doesn't die or breaks the network before the service ends, while at the same time stop new traffic from coming in asap?
Reacted by merussoIts there on https://preliminary.istio.io/latest/docs/reference/commands/pilot-agent/. This is an experimental feature so there is not more docs beyond the generated reference.
Reacted by Laurent Demailly, Jinsu Park and merussoFor future readers here, we implemented this on our Kafka app that would throw 5xx's whenever pods scaled down due to Envoy terminating before the app could finish its outstanding requests. Implemented at 3:25, looks to have fixed it!
Reacted by Luis Iturrios (Koy) and Anthony PoschenWe have a service container that runs a background thread that needs to complete in-flight work and then exit.
The
terminationGracefulPeriodSecondsis used and the service container will exit earlier when the in-flight work completes.Waiting for istio connections to drain does not help with this situation since the background thread doesn't have an open connection. One suggestion I have seen floating around is to have the service container open an otherwise unneeded connection to the istio pod so that istio will wait until the service pod exits. But it feels kind of hacky. Suggestions?
Reacted by 김수겸(Sookyom Kim)/ LastMilehttps://istio.io/latest/blog/2023/native-sidecars/ is the perfect solution. Everything else is pretty rough. 1.29 is available on most platforms these days
Reacted by Jim Kleckner and MaksymThanks! More motivation to get current on k8s. Is there a "least bad" approach in the meantime?
You can use terminationDrainDuration and set it high (works ok but is a static time), or use EXIT_ON_ZERO_ACTIVE_CONNECTIONS (experimental, doesn't work in some cases)
Reacted by Poorna Chandu Gade, Thomas Müller, hayden and siddharthThank you.
There are four (!) settings controlling the sidecar termination duration:
drainDurationis envoy's graceful drain duration (default 45s).parentShutdownDurationis envoy's parent shutdown delay (default 60s).terminationDrainDurationis pilot-agent's termination duration delay after SIGTERM (default 5s).terminationGracefulPeriodSecondsis pod's graceful termination delay after SIGTERM before SIGKILL (default 30s).This is very confusing and is actually inconsistent since (1)-(3) are normally mesh-level and (4) is pod-level. (4) overrides (3), and (3) overrides (1), while (2) is not used. (3) is also too short to be meaningful since envoy's drain is passive.
The proposal is to use (4) as the main deadline since it's orchestrated by kubelet irrespective of mesh-level settings. In this world, there is just one graceful deadline value (4) that dictates how long envoy gracefully drains open connections before shutting down. That means envoy is not terminated abruptly without sending final telemetry, for example. The process would look like this:
cc @ramaraochavali @mandarjog