Using Istio with CronJobs

[Stono]

Hey all,
I have an issue with Istio when used in conjunction with CronJobs or Jobs, in that when the primary pod completes, the "Job" never completes because istio-proxy is still running:

NAME                                  READY     STATUS    RESTARTS   AGE
backup-at-uk-1549872000-7hrx7         1/2       Running   0          34m

I tried adding the following to the end of the primary pod script as suggested by @costinm in #6324, but that doesn't work (envoy exits, proxy doesn't):

curl --max-time 2 -s -f -XPOST http://127.0.0.1:15000/quitquitquit
OK

Which seems to cause envoy to exit correctly, however the istio-proxy process is still running:

istio-proxy@backup-at-uk-1549872000-7hrx7:/$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
istio-p+       1  0.0  0.0  32640 18820 ?        Ssl  08:00   0:00 /usr/local/bin/pilot-agent proxy sidecar --concurrency 1 --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster helm-solr-backup --drainDuration

Despite it no longer listening:

istio-proxy@backup-at-uk-1549872000-7hrx7:/$ netstat -plunt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

The main pod can't send a SIGTERM to istio-proxy because it doesn't have permission to do so (quite rightly) so I'm a little stuck.

The only hacky thing I can think of doing is adding a readinessProbe to istio-proxy which checks to see if it's listening and if it isn't, sends the SIGTERM.

Thoughts?

[Stono]

Associated: kubernetes/kubernetes#25908

[huikang]

Same issue for me.

[Stono]

For those who are interested, we worked around this by adding a livenessProbe to the sidecar injector for istio-proxy:

    livenessProbe:
      exec:
        command:
          - /usr/local/bin/liveness.sh
      initialDelaySeconds: 3
      periodSeconds: 10
      failureThreshold: 5

And then the script looks like this:

#!/bin/bash
set -e 
if ! pidof envoy &>/dev/null; then
  if pidof pilot-agent &>/dev/null; then
    echo "Envoy is not running, exiting istio-proxy"
    kill -s TERM $(pidof pilot-agent)
    exit 1 
  fi 
fi

if ! netstat -plunt | grep 15001; then 
  echo "Istio-proxy not listening on 15001"
  exit 1
fi 
exit 0

[huikang]

@Stono thanks for sharing your work around. Are the above script and the livenessProbe added to the cronjob yams file? I am asking because I could not understand how to adding a livenessProbe to the sidecar injector for istio-proxy. Thanks.

[Stono]

I use the auto sidecar injector. The probes are added to the injected template for the istio-proxy pod. Karl

…

On Mon, 11 Feb 2019, 4:31 pm huikang ***@***.*** wrote: @Stono <https://github.com/Stono> thanks for sharing your work around. Are the above script and the livenessProbe added to the cronjob yams file? I am asking because I could not understand how to adding a livenessProbe to the sidecar injector for istio-proxy. Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABaviQZq0obf_JtZ4xrhya5yXoGshXSqks5vMZrRgaJpZM4azuR_> .

[huikang]

Hi, @Stono, it is still unclear to me how the liveness probe can be added to the istio-proxy pod (my understanding is that the istio-proxy image is not managed by the end user).

Could you point me to some online resources? Thanks.

[Stono]

Ohhh I get you. So in our CI pipeline we build our own istio-proxy image which pulls from the main image and copies these files in. We then use that in the istio sidecar injector Another thing you could do is mount them from a ConfigMap. Karl

…

On Mon, 11 Feb 2019, 5:48 pm huikang ***@***.*** wrote: Hi, @Stono <https://github.com/Stono>, it is still unclear to me how the liveness probe can be added to the istio-proxy pod (my understanding is that the istio-proxy image is not managed by the end user). Could you point me to some online resources? Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABavidP9BXHS5KmBDQs_WxpMVmLW0kAzks5vMazogaJpZM4azuR_> .

[liamawhite]

I would imagine native Istio support for this would require a similar approach. Exposing an additional path on the pilot-agent server that tells it to shut down. This would still require the batch job to call that endpoint after it is finished though.

I wonder if there is something else we could hook into in Kube that would automatically call this endpoint once the job is finished?

[Stono]

@liamawhite do you know of any way to make istio-proxy exit with a 0 status code, at the moment if i send a SIGTERM I get 137, which causes a cronjob failure. I need an exit signal which will shutdown with a 0

[Bessonov]

Similar issue: #11045

[liamawhite]

I think the lastest in release-1.1 should return 0. I will try to find some time to verify.