envoy drain duration should adapt to pod spec #34855

kyessenov · 2021-08-24T23:27:15Z

There are four (!) settings controlling the sidecar termination duration:

drainDuration is envoy's graceful drain duration (default 45s).
parentShutdownDuration is envoy's parent shutdown delay (default 60s).
terminationDrainDuration is pilot-agent's termination duration delay after SIGTERM (default 5s).
terminationGracefulPeriodSeconds is pod's graceful termination delay after SIGTERM before SIGKILL (default 30s).

This is very confusing and is actually inconsistent since (1)-(3) are normally mesh-level and (4) is pod-level. (4) overrides (3), and (3) overrides (1), while (2) is not used. (3) is also too short to be meaningful since envoy's drain is passive.

The proposal is to use (4) as the main deadline since it's orchestrated by kubelet irrespective of mesh-level settings. In this world, there is just one graceful deadline value (4) that dictates how long envoy gracefully drains open connections before shutting down. That means envoy is not terminated abruptly without sending final telemetry, for example. The process would look like this:

On SIGTERM, agent initiates inbound listener graceful drain using (4) as the deadline. GOAWAY and connection:close are sent on all inbound connections, no more inbound connections are accepted, container readiness fails, endpoint removed from the pools. The max drain deadline is set by (4), but envoy can be stopped before that once all outstanding requests/connections are finished (including outbound app-driven). It's probably good to have a fixed minimum delay because detecting if an app (peer container) has outstanding requests is difficult (alternatively, CNI can undo outbound iptables on termination).
After (4) seconds, SIGKILL is issued if envoy does not stop before that.

cc @ramaraochavali @mandarjog

The text was updated successfully, but these errors were encountered:

kyessenov · 2021-08-24T23:27:48Z

Ref envoyproxy/envoy#17056

howardjohn · 2021-08-25T00:19:55Z

We use --drain-strategy immediate which should somewhat help "envoy's drain is passive" if you weren't aware - doesn't work in all cases though.

Overall I agree we should do this. A few concerns I think we will need to work out:

what we should do if users set terminationDrainDuration (or just have the old default of 5s)?
Will we hit any cases where we get stuck? For example, I have seen some users set terminationGracefulPeriodSeconds to extreme values (>10 minutes) - if there are cases where their pod would have shut down and now it doesn't we may run into issues

The benefits largely outweigh the issues, but I we should see what we can do to address those

kyessenov · 2021-08-25T00:29:23Z

@howardjohn It's passive because Envoy does not signal clients pro-actively. On a long-living connection with few requests, clients simply don't receive any notification. For very long grace periods, we have to implement polling for open connections so that if app exits early, proxy does, too. If we do pursue this, I'd imagine flags (1)-(3) are no longer useful. 5s is a dangerously low value, more so for apps with very longs shutdowns.

kyessenov · 2021-08-30T20:00:43Z

One complication is that health checks (and some other endpoints) go through Envoy and those should not count as application connections during drain.

kyessenov · 2021-08-30T20:12:17Z

Tagging @hzxuzhonghu.

ramaraochavali · 2021-09-07T05:29:16Z

One complication is that health checks (and some other endpoints) go through Envoy and those should not count as application connections during drain.

This is exactly the same reason why we can not completely depend on Envoy to provide draining connection count.

kevin-lindsay-1 · 2021-10-08T17:08:52Z

I just ran into this issue; one of my services has a high maximum but low average request time, so it gracefully shuts itself down, but once it's actually exited the sidecar persists and waits around even though there's no active connections.

Hopefully I see this in a release soon, because I'd really like to test it and have even faster resource freeing across our clusters.

ramaraochavali · 2021-10-11T14:44:00Z

@kevin-lindsay-1 #35059 has this implementation. You can set EXIT_ON_ZERO_ACTIVE_CONNECTIONS on Agent side and try it. Please let us know how it works

kevin-lindsay-1 · 2021-10-11T14:47:04Z

@ramaraochavali I use helm to install istio, and can change the image tags. Are there image builds that I could try out?

ramaraochavali · 2021-10-11T14:49:36Z

https://storage.googleapis.com/istio-build/dev/latest - Please try with this

kevin-lindsay-1 · 2021-10-11T16:14:56Z

@ramaraochavali I use dockerhub for my images; can you tell me what I'd need to change on the helm chart for istio-discovery to test that image, because I'm getting an error when trying to switch stuff to that hub/tag.

ramaraochavali · 2021-10-12T05:30:25Z

Sorry. I am not aware of that. May be @howardjohn knows about it

howardjohn · 2021-10-12T17:49:53Z

--set global.hub, --set global.tag

…

On Mon, Oct 11, 2021 at 10:30 PM Rama Chavali ***@***.***> wrote: Sorry. I am not aware of that. May be @howardjohn <https://github.com/howardjohn> knows about it — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34855 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXLMRZ6OZO44EQKOBLLUGPBXXANCNFSM5CX26JXQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

kevin-lindsay-1 · 2021-10-12T22:56:49Z

@howardjohn sorry, I understand the values to set, but I don't know what the actual hub and tag values would be, as this appears to be a google hub and tag, and the attempts I made all gave me errors.

Not experienced with using google storage as a hub, to the point where I'm not even 100% that it's compatible as an implementation of a docker hub.

howardjohn · 2021-10-12T23:42:13Z

gcr.io/istio-testing tag=latest https://github.com/istio/istio/wiki/Dev%20Builds

…

On Tue, Oct 12, 2021 at 3:57 PM Kevin Lindsay ***@***.***> wrote: @howardjohn <https://github.com/howardjohn> sorry, I understand the values to set, but I don't know what the actual hub and tag values would be, as this appears to be a google hub and tag, and the attempts I made all gave me errors. Not experienced with using google storage as a hub, to the point where I'm not even 100% that it's compatible as an implementation of a docker hub. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34855 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXKBF46II7UPDNQ4ARLUGS4LXANCNFSM5CX26JXQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

kevin-lindsay-1 · 2021-10-13T00:01:37Z

Thanks, I'll give it a shot!

kevin-lindsay-1 · 2021-10-13T22:59:15Z

@ramaraochavali I updated the proxyMetada annotation to have EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true', and I see it set in the istio-proxy container, but even though my service has no active connections it still waits the terminationDrainDuration.

Should I not set terminationDrainDuration any longer or something?

Pod logs say version 1.12-alpha.27e28972ff1b1229ddd603ba8363827a3f99fad3-27e28972ff1b1229ddd603ba8363827a3f9

ramaraochavali · 2021-10-14T07:14:58Z

@kevin-lindsay-1 It is not part of proxy metadata. You need to set that as env var while starting proxy.

kevin-lindsay-1 · 2021-10-14T15:27:47Z

@ramaraochavali how do I set that when using the sidecar injector? I tried going in after the pod initialized, but it looks like it doesn't check for that variable after initialization, and I don't see a way to configure the injector to set environment variables that aren't part of a specific pre-defined API.

ramaraochavali · 2021-10-16T07:08:34Z

@kevin-lindsay-1 Sorry. I missed that - since you have already set in proxy Metadata - it should have been carried to env of proxy. Can you please share your pod.yaml and logs?

kevin-lindsay-1 · 2021-10-25T22:24:10Z

@ramaraochavali scratch that. i checked on my config, and I must have tried something previously along the lines of:

proxy.istio.io/proxyMetadata: |
  EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true'

I just tried it again with:

proxy.istio.io/config: |
  proxyMetadata:
    EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true'

and it worked exactly as expected this time.

Sorry for the late response, took a while to get around to testing this again.

ramaraochavali · 2021-10-26T00:52:34Z

Awesome. Thanks for the confirmation

…

On Tue, 26 Oct 2021 at 3:54 AM, Kevin Lindsay ***@***.***> wrote: @ramaraochavali <https://github.com/ramaraochavali> scratch that. i checked on my config, and I must have tried something previously along the lines of: proxy.istio.io/proxyMetadata: | EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true' I just tried it again with: proxy.istio.io/config: | proxyMetadata: EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true' and it worked exactly as expected this time. Sorry for the late response, took a while to get around to testing this again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34855 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEDINKAIJ65INL67XUU7PUDUIXKJLANCNFSM5CX26JXQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

kevin-lindsay-1 · 2021-11-04T20:44:34Z

@ramaraochavali I'd be very interested to know when this feature hits an official release, as it will be very useful for my team, as we use EKS w/ Spot instances, so faster drain time can save us maintenance time and a non-trivial amount of costs over time.

Samjin · 2021-11-04T22:55:17Z

(3) overrides (1), while (2) is not used.

@kyessenov are you saying terminationDrainDuration wouldn't override 1 if 2 is specified?
Assume 3(terminationDrainDuration) is 100s, would envoy shutdown at 100s or does it shutdown at 45s because of drainDuration?
What's the best practice when setting these params? Should 1 and 2 as close to terminationDrainDuration as possible?

ramaraochavali · 2021-11-05T05:51:04Z

It is in 1.12 and here are the dates for https://github.com/istio/istio/wiki/Istio-Release-1.12

Samjin · 2021-11-05T17:42:13Z

@ramaraochavali Thanks for quickly reply. What's the best practice before 1.12?
It sounds to me that 1 and 2 should be a close number to terminationDrainDuration so that envoy doesn't shutdown too early abruptly at 1(45s) before terminationDrainDuration(100s)?

ramaraochavali · 2021-11-08T11:01:30Z

2 is not used now.

Stono · 2021-11-11T20:04:40Z

Hey,
Just catching up on this and the changes in 1.12. Super keen to get this sorted because we currently have a wrapper script for pilot-agent (that we've been using for quite some time) that sort of does this: https://gist.github.com/Stono/2cc68c62968f13e3447fc107abbec72f

If i'm reading the changes right, we could template our workloads with an annotation like this:

annotations:
  proxy.istio.io/config: |
    terminationDrainDuration: <whatever the termination grace period is>
    proxyMetadata:
      EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true'

And should be able to remove our custom wrapper script as that would give me a drain duration === to the termination grace period, and it would exit in a timely manner.

ramaraochavali · 2022-02-27T07:48:46Z

Can you please share Bootstrap's proxyMetadata section if possible?

yteraoka · 2022-02-27T09:52:24Z

jq '.configs[0].bootstrap.node.metadata'

with inclusionRegexps

``` { "ANNOTATIONS": { "kubernetes.io/config.seen": "2022-02-26T22:20:30.119706875+09:00", "kubernetes.io/psp": "eks.privileged", "prometheus.io/scrape": "true", "kubectl.kubernetes.io/default-container": "nginx", "sidecar.istio.io/status": "{\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-data\",\"istio-podinfo\",\"istio-token\",\"istiod-ca-cert\"],\"imagePullSecrets\":null,\"revision\":\"1-12-1\"}", "prometheus.io/path": "/stats/prometheus", "kubernetes.io/config.source": "api", "proxy.istio.io/config": "terminationDrainDuration: 10s\nproxyMetadata:\n EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true'\n MINIMUM_DRAIN_DURATION: 11s\nproxyStatsMatcher:\n inclusionRegexps:\n - \".*downstream_cx_active\"\n", "prometheus.io/port": "15020", "kubectl.kubernetes.io/default-logs-container": "nginx" }, "WORKLOAD_NAME": "nginx", "INTERCEPTION_MODE": "REDIRECT", "ENVOY_STATUS_PORT": 15021, "NAME": "nginx-55475b47b6-57m8z", "INSTANCE_IPS": "10.110.101.240", "ISTIO_PROXY_SHA": "istio-proxy:e6f45abcf874983fbff384459d70b28c072f68b5", "CLUSTER_ID": "Kubernetes", "OWNER": "kubernetes://apis/apps/v1/namespaces/teraoka/deployments/nginx", "LABELS": { "pod-template-hash": "55475b47b6", "service.istio.io/canonical-revision": "latest", "app.kubernetes.io/name": "nginx", "security.istio.io/tlsMode": "istio", "app.kubernetes.io/instance": "nginx", "service.istio.io/canonical-name": "nginx" }, "ISTIO_VERSION": "1.12.1", "SERVICE_ACCOUNT": "nginx", "PILOT_SAN": [ "istiod-1-12-1.istio-system.svc" ], "PROXY_CONFIG": { "proxyMetadata": { "MINIMUM_DRAIN_DURATION": "11s", "EXIT_ON_ZERO_ACTIVE_CONNECTIONS": "true" }, "controlPlaneAuthPolicy": "MUTUAL_TLS", "binaryPath": "/usr/local/bin/envoy", "proxyStatsMatcher": { "inclusionRegexps": [ ".*downstream_cx_active" ] }, "configPath": "./etc/istio/proxy", "proxyAdminPort": 15000, "statNameLength": 189, "discoveryAddress": "istiod-1-12-1.istio-system.svc:15012", "drainDuration": "45s", "statusPort": 15020, "terminationDrainDuration": "10s", "concurrency": 2, "tracing": { "zipkin": { "address": "zipkin.istio-system:9411" } }, "holdApplicationUntilProxyStarts": true, "parentShutdownDuration": "60s", "serviceCluster": "istio-proxy" }, "MESH_ID": "cluster.local", "APP_CONTAINERS": "nginx", "NAMESPACE": "teraoka", "PROV_CERT": "var/run/secrets/istio/root-cert.pem", "ENVOY_PROMETHEUS_PORT": 15090, "POD_PORTS": "[{\"name\":\"http\",\"containerPort\":80,\"protocol\":\"TCP\"}]" } ```

without inclusionRegexps

{
  "APP_CONTAINERS": "nginx",
  "PROV_CERT": "var/run/secrets/istio/root-cert.pem",
  "NAMESPACE": "teraoka",
  "POD_PORTS": "[{\"name\":\"http\",\"containerPort\":80,\"protocol\":\"TCP\"}]",
  "ANNOTATIONS": {
    "kubernetes.io/config.seen": "2022-02-27T15:32:04.461976770+09:00",
    "proxy.istio.io/config": "terminationDrainDuration: 10s\nproxyMetadata:\n  EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true'\n  MINIMUM_DRAIN_DURATION: 11s\n",
    "kubectl.kubernetes.io/default-logs-container": "nginx",
    "kubernetes.io/psp": "eks.privileged",
    "kubernetes.io/config.source": "api",
    "prometheus.io/scrape": "true",
    "sidecar.istio.io/status": "{\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-data\",\"istio-podinfo\",\"istio-token\",\"istiod-ca-cert\"],\"imagePullSecrets\":null,\"revision\":\"1-12-1\"}",
    "prometheus.io/port": "15020",
    "kubectl.kubernetes.io/default-container": "nginx",
    "prometheus.io/path": "/stats/prometheus"
  },
  "ISTIO_VERSION": "1.12.1",
  "LABELS": {
    "app.kubernetes.io/instance": "nginx",
    "service.istio.io/canonical-revision": "latest",
    "service.istio.io/canonical-name": "nginx",
    "app.kubernetes.io/name": "nginx",
    "pod-template-hash": "84c877988c",
    "security.istio.io/tlsMode": "istio"
  },
  "CLUSTER_ID": "Kubernetes",
  "PROXY_CONFIG": {
    "drainDuration": "45s",
    "parentShutdownDuration": "60s",
    "configPath": "./etc/istio/proxy",
    "tracing": {
      "zipkin": {
        "address": "zipkin.istio-system:9411"
      }
    },
    "binaryPath": "/usr/local/bin/envoy",
    "concurrency": 2,
    "proxyAdminPort": 15000,
    "serviceCluster": "istio-proxy",
    "statNameLength": 189,
    "statusPort": 15020,
    "discoveryAddress": "istiod-1-12-1.istio-system.svc:15012",
    "holdApplicationUntilProxyStarts": true,
    "terminationDrainDuration": "10s",
    "proxyMetadata": {
      "MINIMUM_DRAIN_DURATION": "11s",
      "EXIT_ON_ZERO_ACTIVE_CONNECTIONS": "true"
    },
    "controlPlaneAuthPolicy": "MUTUAL_TLS"
  },
  "ENVOY_STATUS_PORT": 15021,
  "OWNER": "kubernetes://apis/apps/v1/namespaces/teraoka/deployments/nginx",
  "PILOT_SAN": [
    "istiod-1-12-1.istio-system.svc"
  ],
  "NAME": "nginx-84c877988c-s9vd4",
  "WORKLOAD_NAME": "nginx",
  "INTERCEPTION_MODE": "REDIRECT",
  "ISTIO_PROXY_SHA": "istio-proxy:e6f45abcf874983fbff384459d70b28c072f68b5",
  "MESH_ID": "cluster.local",
  "ENVOY_PROMETHEUS_PORT": 15090,
  "INSTANCE_IPS": "10.110.103.52",
  "SERVICE_ACCOUNT": "nginx"
}

diff

$ diff -u <(jq -S '.configs[0].bootstrap.node.metadata' config.dump.with-inclusion.json) <(jq -S '.configs[0].bootstrap.node.metadata' config.dump.without-inclusion.json)
--- /dev/fd/11	2022-02-27 18:49:32.000000000 +0900
+++ /dev/fd/12	2022-02-27 18:49:32.000000000 +0900
@@ -2,33 +2,33 @@
   "ANNOTATIONS": {
     "kubectl.kubernetes.io/default-container": "nginx",
     "kubectl.kubernetes.io/default-logs-container": "nginx",
-    "kubernetes.io/config.seen": "2022-02-26T22:20:30.119706875+09:00",
+    "kubernetes.io/config.seen": "2022-02-27T15:32:04.461976770+09:00",
     "kubernetes.io/config.source": "api",
     "kubernetes.io/psp": "eks.privileged",
     "prometheus.io/path": "/stats/prometheus",
     "prometheus.io/port": "15020",
     "prometheus.io/scrape": "true",
-    "proxy.istio.io/config": "terminationDrainDuration: 10s\nproxyMetadata:\n  EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true'\n  MINIMUM_DRAIN_DURATION: 11s\nproxyStatsMatcher:\n  inclusionRegexps:\n    - \".*downstream_cx_active\"\n",
+    "proxy.istio.io/config": "terminationDrainDuration: 10s\nproxyMetadata:\n  EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true'\n  MINIMUM_DRAIN_DURATION: 11s\n",
     "sidecar.istio.io/status": "{\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-data\",\"istio-podinfo\",\"istio-token\",\"istiod-ca-cert\"],\"imagePullSecrets\":null,\"revision\":\"1-12-1\"}"
   },
   "APP_CONTAINERS": "nginx",
   "CLUSTER_ID": "Kubernetes",
   "ENVOY_PROMETHEUS_PORT": 15090,
   "ENVOY_STATUS_PORT": 15021,
-  "INSTANCE_IPS": "10.110.101.240",
+  "INSTANCE_IPS": "10.110.103.52",
   "INTERCEPTION_MODE": "REDIRECT",
   "ISTIO_PROXY_SHA": "istio-proxy:e6f45abcf874983fbff384459d70b28c072f68b5",
   "ISTIO_VERSION": "1.12.1",
   "LABELS": {
     "app.kubernetes.io/instance": "nginx",
     "app.kubernetes.io/name": "nginx",
-    "pod-template-hash": "55475b47b6",
+    "pod-template-hash": "84c877988c",
     "security.istio.io/tlsMode": "istio",
     "service.istio.io/canonical-name": "nginx",
     "service.istio.io/canonical-revision": "latest"
   },
   "MESH_ID": "cluster.local",
-  "NAME": "nginx-55475b47b6-57m8z",
+  "NAME": "nginx-84c877988c-s9vd4",
   "NAMESPACE": "teraoka",
   "OWNER": "kubernetes://apis/apps/v1/namespaces/teraoka/deployments/nginx",
   "PILOT_SAN": [
@@ -50,11 +50,6 @@
       "EXIT_ON_ZERO_ACTIVE_CONNECTIONS": "true",
       "MINIMUM_DRAIN_DURATION": "11s"
     },
-    "proxyStatsMatcher": {
-      "inclusionRegexps": [
-        ".*downstream_cx_active"
-      ]
-    },
     "serviceCluster": "istio-proxy",
     "statNameLength": 189,
     "statusPort": 15020,

ramaraochavali · 2022-02-27T10:56:36Z

@yteraoka Thank you. I see the problem. Fixed in #37573

yteraoka · 2022-02-27T11:58:04Z

@ramaraochavali Thank you, too.

kevin-lindsay-1 · 2022-05-12T15:28:55Z

Does this actually work now? If so, when can I test it in a non-dev build?

liyihuang · 2022-05-23T03:41:01Z

@ramaraochavali I have seen your PR back in Feb. I'm on 1.13.4 and still don't see this has been merged and only see it on 1.14 beta will your 37573 be merged to any 1.12 or 1.13?

2022-05-23T02:54:09.595581Z     info    Envoy proxy is ready
2022-05-23T02:58:22.079931Z     info    Agent draining Proxy
2022-05-23T02:58:22.079917Z     info    Status server has successfully terminated
2022-05-23T02:58:22.080142Z     error   accept tcp [::]:15020: use of closed network connection
2022-05-23T02:58:22.081261Z     info    Agent draining proxy for 5s, then waiting for active connections to terminate...
2022-05-23T02:58:27.082178Z     info    Checking for active connections...
2022-05-23T02:58:28.085056Z     info    There are still -1 active connections
2022-05-23T02:58:29.084947Z     info    There are still -1 active connections
2022-05-23T02:58:30.084276Z     info    There are still -1 active connections
2022-05-23T02:58:31.085118Z     info    There are still -1 active connections
2022-05-23T02:58:32.084250Z     info    There are still -1 active connections
2022-05-23T02:58:33.084492Z     info    There are still -1 active connections

my IOP spec

   meshConfig:
     defaultConfig:
       terminationDrainDuration: 150s
       proxyMetadata:
         EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true'`
     enablePrometheusMerge: true

ramaraochavali · 2022-05-23T04:03:40Z

I have added for 1.13. Please follow this PR #39082

liyihuang · 2022-05-24T15:29:42Z

@ramaraochavali thanks. can we also cherry-pick this for 1.12?

ramaraochavali · 2022-05-24T15:50:32Z

1.12 needs manual PR. It will take some time.

kevin-lindsay-1 · 2022-05-24T16:05:21Z

@ramaraochavali thanks! can you let us know when this will be in a release build? i've sometimes seen merges take months to see in a release build.

ramaraochavali · 2022-06-05T05:05:42Z

Sorry I have been busy with other stuff. PR for backport #39286 to 1.12

istio-policy-bot · 2022-09-18T05:07:25Z

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2022-06-05. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

batchamalick · 2022-11-17T05:35:18Z

Hi @ramaraochavali ,
While testing the setting EXIT_ON_ZERO_ACTIVE_CONNECTIONS: 'true' ,I deliberately kept an active connection and killed the pod using kubectl delete ,the envoy logs says one active connection clearly but after the terminationGracefulPeriod the istio-proxy + app is killed with active connection, Should there be a doc to say that user needs to adjust the terminationGracefulPeriod if the app takes more time than default 30s.

ramaraochavali · 2022-11-17T08:45:59Z

Sure. We can update the docs

ldemailly · 2022-12-09T18:11:56Z

EXIT_ON_ZERO_ACTIVE_CONNECTIONS isn't mentioned in latest docs - has the default changed to true or it still exists in current istio as an option? and maybe related is there an uptodate doc on how to ensure the proxy doesn't die or breaks the network before the service ends, while at the same time stop new traffic from coming in asap?

howardjohn · 2022-12-09T18:13:02Z

Its there on https://preliminary.istio.io/latest/docs/reference/commands/pilot-agent/. This is an experimental feature so there is not more docs beyond the generated reference.

jbilliau-rcd · 2023-02-03T15:08:00Z

For future readers here, we implemented this on our Kafka app that would throw 5xx's whenever pods scaled down due to Envoy terminating before the app could finish its outstanding requests. Implemented at 3:25, looks to have fixed it!

jkleckner · 2024-01-26T22:09:13Z

We have a service container that runs a background thread that needs to complete in-flight work and then exit.

The terminationGracefulPeriodSeconds is used and the service container will exit earlier when the in-flight work completes.

Waiting for istio connections to drain does not help with this situation since the background thread doesn't have an open connection. One suggestion I have seen floating around is to have the service container open an otherwise unneeded connection to the istio pod so that istio will wait until the service pod exits. But it feels kind of hacky. Suggestions?

howardjohn · 2024-01-26T22:12:11Z

https://istio.io/latest/blog/2023/native-sidecars/ is the perfect solution. Everything else is pretty rough. 1.29 is available on most platforms these days

jkleckner · 2024-01-30T15:25:47Z

Thanks! More motivation to get current on k8s. Is there a "least bad" approach in the meantime?

howardjohn · 2024-01-30T16:25:57Z

You can use terminationDrainDuration and set it high (works ok but is a static time), or use EXIT_ON_ZERO_ACTIVE_CONNECTIONS (experimental, doesn't work in some cases)

jkleckner · 2024-01-31T04:40:07Z

Thank you.

This was referenced Sep 7, 2021

terminate envoy when number of active connections is zero #35043

Closed

terminate envoy when number of active connections is zero #35059

Merged

ramaraochavali mentioned this issue Feb 27, 2022

fix passing exit on zero active connections to metadata #37573

Merged

10 tasks

skonto mentioned this issue Mar 22, 2022

[SRVKS-893] Do not apply serving-tests namespace when already exist openshift-knative/serverless-operator#1443

Closed

istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Sep 3, 2022

joshuaswickirl mentioned this issue Sep 13, 2022

add support for drain and quit envoyproxy/envoy#17056

Closed

istio-policy-bot closed this as completed Sep 18, 2022

istio-policy-bot added the lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. label Sep 18, 2022

jbilliau-rcd mentioned this issue Dec 1, 2022

Envoy shutting down before the thing it's wrapping can cause failed requests #7136

Closed

ramaraochavali mentioned this issue Sep 26, 2023

Does default config for TerminationDrainDuration should greater than DrainDuration? #47117

Closed

envoy drain duration should adapt to pod spec #34855

envoy drain duration should adapt to pod spec #34855

Comments

kyessenov commented Aug 24, 2021 • edited Loading

kyessenov commented Aug 24, 2021

howardjohn commented Aug 25, 2021

kyessenov commented Aug 25, 2021 • edited Loading

kyessenov commented Aug 30, 2021

kyessenov commented Aug 30, 2021

ramaraochavali commented Sep 7, 2021

kevin-lindsay-1 commented Oct 8, 2021

ramaraochavali commented Oct 11, 2021

kevin-lindsay-1 commented Oct 11, 2021

ramaraochavali commented Oct 11, 2021

kevin-lindsay-1 commented Oct 11, 2021 • edited Loading

ramaraochavali commented Oct 12, 2021

howardjohn commented Oct 12, 2021 via email

kevin-lindsay-1 commented Oct 12, 2021

howardjohn commented Oct 12, 2021 via email

kevin-lindsay-1 commented Oct 13, 2021

kevin-lindsay-1 commented Oct 13, 2021 • edited Loading

ramaraochavali commented Oct 14, 2021

kevin-lindsay-1 commented Oct 14, 2021

ramaraochavali commented Oct 16, 2021

kevin-lindsay-1 commented Oct 25, 2021 • edited Loading

ramaraochavali commented Oct 26, 2021 via email

kevin-lindsay-1 commented Nov 4, 2021

Samjin commented Nov 4, 2021 • edited Loading

ramaraochavali commented Nov 5, 2021

Samjin commented Nov 5, 2021 • edited Loading

ramaraochavali commented Nov 8, 2021

Stono commented Nov 11, 2021

ramaraochavali commented Feb 27, 2022

yteraoka commented Feb 27, 2022

ramaraochavali commented Feb 27, 2022

yteraoka commented Feb 27, 2022

kevin-lindsay-1 commented May 12, 2022

liyihuang commented May 23, 2022

ramaraochavali commented May 23, 2022

liyihuang commented May 24, 2022

ramaraochavali commented May 24, 2022

kevin-lindsay-1 commented May 24, 2022 • edited Loading

ramaraochavali commented Jun 5, 2022

istio-policy-bot commented Sep 18, 2022

batchamalick commented Nov 17, 2022

ramaraochavali commented Nov 17, 2022

ldemailly commented Dec 9, 2022

howardjohn commented Dec 9, 2022

jbilliau-rcd commented Feb 3, 2023

jkleckner commented Jan 26, 2024

howardjohn commented Jan 26, 2024

jkleckner commented Jan 30, 2024

howardjohn commented Jan 30, 2024

jkleckner commented Jan 31, 2024

kyessenov commented Aug 24, 2021 •

edited

Loading

kyessenov commented Aug 25, 2021 •

edited

Loading

kevin-lindsay-1 commented Oct 11, 2021 •

edited

Loading

kevin-lindsay-1 commented Oct 13, 2021 •

edited

Loading

kevin-lindsay-1 commented Oct 25, 2021 •

edited

Loading

Samjin commented Nov 4, 2021 •

edited

Loading

Samjin commented Nov 5, 2021 •

edited

Loading

kevin-lindsay-1 commented May 24, 2022 •

edited

Loading