Using Istio with CronJobs #11659

Stono · 2019-02-11T08:37:06Z

Hey all,
I have an issue with Istio when used in conjunction with CronJobs or Jobs, in that when the primary pod completes, the "Job" never completes because istio-proxy is still running:

NAME                                  READY     STATUS    RESTARTS   AGE
backup-at-uk-1549872000-7hrx7         1/2       Running   0          34m

I tried adding the following to the end of the primary pod script as suggested by @costinm in #6324, but that doesn't work (envoy exits, proxy doesn't):

curl --max-time 2 -s -f -XPOST http://127.0.0.1:15000/quitquitquit
OK

Which seems to cause envoy to exit correctly, however the istio-proxy process is still running:

istio-proxy@backup-at-uk-1549872000-7hrx7:/$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
istio-p+       1  0.0  0.0  32640 18820 ?        Ssl  08:00   0:00 /usr/local/bin/pilot-agent proxy sidecar --concurrency 1 --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster helm-solr-backup --drainDuration

Despite it no longer listening:

istio-proxy@backup-at-uk-1549872000-7hrx7:/$ netstat -plunt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

The main pod can't send a SIGTERM to istio-proxy because it doesn't have permission to do so (quite rightly) so I'm a little stuck.

The only hacky thing I can think of doing is adding a readinessProbe to istio-proxy which checks to see if it's listening and if it isn't, sends the SIGTERM.

Thoughts?

The text was updated successfully, but these errors were encountered:

Stono · 2019-02-11T10:30:56Z

Associated: kubernetes/kubernetes#25908

huikang · 2019-02-11T14:56:18Z

Same issue for me.

Stono · 2019-02-11T15:15:27Z

For those who are interested, we worked around this by adding a livenessProbe to the sidecar injector for istio-proxy:

    livenessProbe:
      exec:
        command:
          - /usr/local/bin/liveness.sh
      initialDelaySeconds: 3
      periodSeconds: 10
      failureThreshold: 5

And then the script looks like this:

#!/bin/bash
set -e 
if ! pidof envoy &>/dev/null; then
  if pidof pilot-agent &>/dev/null; then
    echo "Envoy is not running, exiting istio-proxy"
    kill -s TERM $(pidof pilot-agent)
    exit 1 
  fi 
fi

if ! netstat -plunt | grep 15001; then 
  echo "Istio-proxy not listening on 15001"
  exit 1
fi 
exit 0

huikang · 2019-02-11T16:30:42Z

@Stono thanks for sharing your work around. Are the above script and the livenessProbe added to the cronjob yams file? I am asking because I could not understand how to adding a livenessProbe to the sidecar injector for istio-proxy. Thanks.

Stono · 2019-02-11T16:33:11Z

I use the auto sidecar injector. The probes are added to the injected template for the istio-proxy pod. Karl

…

On Mon, 11 Feb 2019, 4:31 pm huikang ***@***.*** wrote: @Stono <https://github.com/Stono> thanks for sharing your work around. Are the above script and the livenessProbe added to the cronjob yams file? I am asking because I could not understand how to adding a livenessProbe to the sidecar injector for istio-proxy. Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABaviQZq0obf_JtZ4xrhya5yXoGshXSqks5vMZrRgaJpZM4azuR_> .

huikang · 2019-02-11T17:47:56Z

Hi, @Stono, it is still unclear to me how the liveness probe can be added to the istio-proxy pod (my understanding is that the istio-proxy image is not managed by the end user).

Could you point me to some online resources? Thanks.

Stono · 2019-02-11T17:54:09Z

Ohhh I get you. So in our CI pipeline we build our own istio-proxy image which pulls from the main image and copies these files in. We then use that in the istio sidecar injector Another thing you could do is mount them from a ConfigMap. Karl

…

On Mon, 11 Feb 2019, 5:48 pm huikang ***@***.*** wrote: Hi, @Stono <https://github.com/Stono>, it is still unclear to me how the liveness probe can be added to the istio-proxy pod (my understanding is that the istio-proxy image is not managed by the end user). Could you point me to some online resources? Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABavidP9BXHS5KmBDQs_WxpMVmLW0kAzks5vMazogaJpZM4azuR_> .

liamawhite · 2019-02-12T09:55:37Z

I would imagine native Istio support for this would require a similar approach. Exposing an additional path on the pilot-agent server that tells it to shut down. This would still require the batch job to call that endpoint after it is finished though.

I wonder if there is something else we could hook into in Kube that would automatically call this endpoint once the job is finished?

Stono · 2019-02-12T14:23:12Z

@liamawhite do you know of any way to make istio-proxy exit with a 0 status code, at the moment if i send a SIGTERM I get 137, which causes a cronjob failure. I need an exit signal which will shutdown with a 0

Bessonov · 2019-02-13T16:43:21Z

Similar issue: #11045

liamawhite · 2019-02-14T07:04:22Z

I think the lastest in release-1.1 should return 0. I will try to find some time to verify.

Stono · 2019-04-03T15:45:42Z

Seems to be fine for me in 1.1.1

Stono · 2019-04-03T15:47:42Z

We have started doing this recently btw folks in our pod spec:

            command: ["/bin/bash", "-c"]
            args:
              - |
                trap "curl --max-time 2 -s -f -XPOST http://127.0.0.1:15000/quitquitquit" EXIT
                while ! curl -s -f http://127.0.0.1:15020/healthz/ready; do sleep 1; done
                sleep 2
                {{ $.Values.cron.command }}

This will:

Wait for proxy to be ready
Tell it to quit when done

mikesimons · 2019-04-30T10:32:40Z

@Stono Have you observed the sidecar living for a while after receiving quitquitquit? Ours are living for another minute or so before exiting (although 15000 gets closed immediately).

We're also getting log spam on Completed jobs until we delete them:

info	Envoy proxy is NOT ready: failed retrieving Envoy stats: Get http://127.0.0.1:15000/stats?usedonly: dial tcp 127.0.0.1:15000: connect: connection refused

Stono · 2019-04-30T10:34:37Z

Yeah so we also have a liveness probe that looks for 15000 listening and if it's missing exits. It's hacky.

…

On Tue, 30 Apr 2019, 11:33 am Mike Simons, ***@***.***> wrote: @Stono <https://github.com/Stono> Have you observed the sidecar living for a while after receiving quitquitquit? Ours are living for another minute or so before exiting (although 15000 gets closed immediately). We're also getting log spam on Completed jobs until we delete them: info Envoy proxy is NOT ready: failed retrieving Envoy stats: Get http://127.0.0.1:15000/stats?usedonly: dial tcp 127.0.0.1:15000: connect: connection refused — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALK7CKBTH2COP3SKOIRR2LPTAN7JANCNFSM4GWO4R7Q> .

stale · 2019-07-29T10:44:17Z

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

Bessonov · 2019-07-31T18:52:07Z

Activity!

howardjohn · 2019-09-22T22:13:37Z

In 1.3 we added a new /quitquitquit endpoint to pilot agent which should resolve this. Its not perfect since it requires some manual action, but I think a longer term solution would depend on kubernetes/kubernetes#65502 or #11366 or similar. If there is anything else we can do to improve these in the short term I would be happy to take a look at

rdavyd · 2022-01-21T19:28:03Z

@howardjohn Did some digging and testing and did not find a way to do such a tweak. Looks like k8s takes the displayed PODStatus from one of the containers. Tested POD with 2 containers and without istio-proxy and its PODStatus always returned the exit status of the first container in the array.
But when adding istio-proxy (it adds as the second container) this behavior changes and PODStatus displays the exit status of istio-proxy.
Also didn't find envoy API call like /quitquitquit that forces exit with non zero status.

Update: I think I got to the bottom of this. The resulting POD Status display message is calculated when returning to client. And in our case it is unstable and sometimes wrong. It returns the termination reason of the last container in the pod.Status.ContainerStatuses array. I don't think this array sequence is controlled by user.

https://github.com/kubernetes/kubernetes/blob/5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e/pkg/printers/internalversion/printers.go#L812-L813

The link is for version 1.22.5, but in master I see the same.

sathieu · 2022-02-08T12:07:05Z

Hello, I've done an "operator" to handle this problem until keystone containers are added to k8s.

See: https://gitlab.com/kubitus-project/kubitus-pod-cleaner-operator/-/blob/main/README.md

ceastman-r7 · 2022-03-16T20:42:13Z

This seems to maintain the error code from the job:

x=$(echo $?); curl -fsI -X POST http://localhost:15020/quitquitquit || true && exit $x

andrewhibbert · 2022-06-01T12:42:29Z

Seems it is not always the same port. The following works for me

x=(echo $?); curl -fsI -X POST http://localhost:15000/quitquitquit && exit $x

jacekszlachtass · 2022-06-06T07:00:05Z

dollar sign is missing in the above command:

x=$(echo $?); curl -fsI -X POST http://localhost:15000/quitquitquit && exit $x

kschu91 · 2022-07-15T07:51:08Z

Just if anyone is searching for a simple solution: https://github.com/AOEpeople/kubernetes-sidecar-cleaner

We have developed a simple app to clean up the istio-proxies in completed CronJobs.

ZiaUrRehman-GBI · 2022-10-22T15:56:22Z

@kschu91 where should be this controller install? Inside istio namespace?

brunocascio · 2023-02-08T15:52:04Z

Is there any plan to implement something at istio level rather than using hacky solutions or a 3rd party operator?

mmerickel · 2023-02-08T16:01:43Z

https://istio.io/latest/blog/2022/introducing-ambient-mesh/ can't come soon enough. :-)

ceastman-r7 · 2023-02-08T17:48:03Z

There is a kubernetes enhancement that should allow orderly startup and shutdown of multi container pods: kubernetes/enhancements#3759

or ambient :)

nickzelei · 2023-02-09T18:32:47Z

Just if anyone is searching for a simple solution: https://github.com/AOEpeople/kubernetes-sidecar-cleaner

We have developed a simple app to clean up the istio-proxies in completed CronJobs.

This worked perfectly for me - thank you for your contribution!

@ZiaUrRehman-GBI - you can deploy it anywhere. The helmchart is deployed with a cluster role so it moderates pods in the entire cluster. At least that's how it works with the helmchart that @kschu91 is providing.
You'd have to roll your own deployment if you wanted it to be scoped to a specific namespace.

brunocascio · 2023-02-09T18:41:03Z

So there's no official solution.

I prefer to use a script to notify Istio rather than installing an unofficial or hacky operator 😃

Thanks everyone

afirth · 2023-08-30T09:23:58Z

This should be addressed by https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/#what-are-sidecar-containers-in-1-28

bmitchinson · 2024-01-17T20:34:03Z

Not a solution by any means, but a reminder that you can opt out of istio when appropriate by using the sidecar.istio.io/inject: "false" pod annotation.

yasensim · 2024-01-18T05:10:10Z

Sidecar as a first class citizen is in alpha in behind a feature gate in Kubernetes 1.28 and is beta and enabled by default in Kubernetes 1.29. If/when Istio implements restartPolicy the istio-proxy lifecycle will align to the main container. This will allow running Jobs on Istio.
For reference HERE .

cc: @howardjohn

howardjohn · 2024-01-18T15:20:58Z

Istio already supports it behind the ENABLE_NATIVE_SIDECARS. Note I highly recommend only using it on K8s 1.29 as some of the lifecycle is broken in 1.28

kmushegi · 2024-05-25T17:02:10Z

Just if anyone is searching for a simple solution: https://github.com/AOEpeople/kubernetes-sidecar-cleaner

We have developed a simple app to clean up the istio-proxies in completed CronJobs.

This is exactly what I needed, thank you!!

@kschu91 where should be this controller install? Inside istio namespace?

I installed it into the istio-system namespace since the watcher watches all namespaces: https://github.com/AOEpeople/kubernetes-sidecar-cleaner/blob/main/main.go#L54 and runs with a clusterrole

Stono · 2024-08-24T18:44:24Z

I'm going to close this supperrrr old issue.
Native sidecars is the solution (albeit some gotchas i've talked about here https://karlstoney.com/moving-to-native-sidecars/)

Callek · 2024-08-28T03:39:20Z

I'm going to close this supperrrr old issue. Native sidecars is the solution (albeit some gotchas i've talked about here https://karlstoney.com/moving-to-native-sidecars/)

@Stono from that linked article they actually recommend not using it with istio:

Bit of an update here; I wouldn't do it with Istio just yet! There's a reasonably nasty bug I've stumbled across which means outbound connections from your app will start seeing Connection: close. This has pretty horrid interactions with some HTTP connection pools.

Stono · 2024-08-28T08:11:35Z

I know, I wrote it :)

My point was more that there won't be a "solution" to this multi year problem without sidecar containers, the actual solution is Sidecar containers. So the choices are either:

Continue with hacky script work arounds (which work, i used them for years)
Move to Sidecar containers, but be aware of those couple of gotchas I mentioned in the blog. I've detailed my mitigation to the shutdown issue here Native Sidecars draining erroneously sending connection: close for outbound requests #51855 which seems to work fine for me, i've been using Sidecar containers in production for a while now.

Either approach currently has some daemons, but Sidecar containers is less hacky and moving in the correct direction so will only get better with time.

ruslanguns · 2024-08-28T12:17:28Z

Hi @Stono, thanks for your article! I’m still a bit confused about the best way to set this up in a cron job. Could you provide some guidance on how to properly configure the startupProbe with Istio Proxy? What would be the best approach for this?

Stono · 2024-08-28T12:19:46Z

Use sidecar containers, they start before your app.

…

On Wed, 28 Aug 2024, 13:17 Ruslan Gonzalez, ***@***.***> wrote: Hi @Stono <https://github.com/Stono>, thanks for your article! I’m still a bit confused about the best way to set this up in a cron job. Could you provide some guidance on how to properly configure the startupProbe with Istio Proxy? What would be the best approach for this? — Reply to this email directly, view it on GitHub <#11659 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALK7CJEBNYHBNVUIFX2KT3ZTW5XHAVCNFSM4GWO4R72U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMZRGUYTMNRXG4YA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Stono mentioned this issue Feb 11, 2019

Cronjob run incorrectly with istio #11657

Closed

Stono mentioned this issue Feb 11, 2019

Better support for sidecar containers in batch jobs #6324

Closed

rshriram added the area/networking label Feb 11, 2019

rshriram assigned liamawhite Feb 11, 2019

Stono closed this as completed Apr 3, 2019

Stono reopened this Apr 3, 2019

Stono mentioned this issue Jun 20, 2019

Pilot-agent to implement a quitquitquit endpoint like envoy #15041

Closed

stale bot added the stale label Jul 29, 2019

stale bot removed the stale label Jul 31, 2019

esnible mentioned this issue Aug 1, 2019

Job ttlSecondsAfterFinished is not working with Istio #15873

Closed

howardjohn closed this as completed Sep 22, 2019

howardjohn mentioned this issue Sep 24, 2019

istio/tools - replace config changer with CronJob #17353

Closed

rdavyd mentioned this issue Jan 24, 2022

Inconsistent POD status reporting kubernetes/kubernetes#107713

Open

DoGab mentioned this issue Dec 5, 2022

feat: use k8s init and setup jobs and remove cockroach chart dependency zitadel/zitadel-charts#48

Merged

ymesika mentioned this issue Jan 16, 2023

Envoy container prevents job from ending #42824

Closed

ceastman-r7 mentioned this issue Mar 20, 2023

Terminate pod on container completion kubernetes/enhancements#3582

Open

4 tasks

loujar mentioned this issue Jul 26, 2023

executors: Support for Container Lifecycle Hooks for Native k8s Executor jobs sourcegraph/sourcegraph-public-snapshot#55338

Open

wleese mentioned this issue Sep 14, 2023

Adopt Scuttle #46991

Closed

yasensim mentioned this issue Jan 18, 2024

Consider automatic ENABLE_NATIVE_SIDECARS support #48794

Open

Stono closed this as completed Aug 24, 2024

Using Istio with CronJobs #11659

Using Istio with CronJobs #11659

Comments

Stono commented Feb 11, 2019 • edited Loading

Stono commented Feb 11, 2019

huikang commented Feb 11, 2019

Stono commented Feb 11, 2019

huikang commented Feb 11, 2019

Stono commented Feb 11, 2019 via email

huikang commented Feb 11, 2019

Stono commented Feb 11, 2019 via email • edited Loading

liamawhite commented Feb 12, 2019 • edited Loading

Stono commented Feb 12, 2019

Bessonov commented Feb 13, 2019

liamawhite commented Feb 14, 2019

Stono commented Apr 3, 2019

Stono commented Apr 3, 2019 • edited Loading

mikesimons commented Apr 30, 2019

Stono commented Apr 30, 2019 via email

stale bot commented Jul 29, 2019

Bessonov commented Jul 31, 2019

howardjohn commented Sep 22, 2019

rdavyd commented Jan 21, 2022 • edited Loading

sathieu commented Feb 8, 2022

ceastman-r7 commented Mar 16, 2022 • edited Loading

andrewhibbert commented Jun 1, 2022

jacekszlachtass commented Jun 6, 2022

kschu91 commented Jul 15, 2022

ZiaUrRehman-GBI commented Oct 22, 2022

brunocascio commented Feb 8, 2023 • edited Loading

mmerickel commented Feb 8, 2023

ceastman-r7 commented Feb 8, 2023

nickzelei commented Feb 9, 2023 • edited Loading

brunocascio commented Feb 9, 2023

afirth commented Aug 30, 2023

bmitchinson commented Jan 17, 2024

yasensim commented Jan 18, 2024

howardjohn commented Jan 18, 2024

kmushegi commented May 25, 2024

Stono commented Aug 24, 2024

Callek commented Aug 28, 2024

Stono commented Aug 28, 2024 • edited Loading

ruslanguns commented Aug 28, 2024

Stono commented Aug 28, 2024 via email

Stono commented Feb 11, 2019 •

edited

Loading

Stono commented Feb 11, 2019 via email •

edited

Loading

liamawhite commented Feb 12, 2019 •

edited

Loading

Stono commented Apr 3, 2019 •

edited

Loading

rdavyd commented Jan 21, 2022 •

edited

Loading

ceastman-r7 commented Mar 16, 2022 •

edited

Loading

brunocascio commented Feb 8, 2023 •

edited

Loading

nickzelei commented Feb 9, 2023 •

edited

Loading

Stono commented Aug 28, 2024 •

edited

Loading