webhook initialization fails with istio injection enabled

[tony-kerz]

when trying to deploy the helm chart into a namespace (in a cluster with istio installed) with injection enabled
(there is a label on the namespace like istio-injection=enabled), the {blah}-admission-create will seem to complete, with the following in the logs:

2020-10-29T21:51:36.510353454Z W1029 21:51:36.510113       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-10-29T21:51:36.522315230Z {"level":"info","msg":"secret already exists","source":"cmd/create.go:28","time":"2020-10-29T21:51:36Z"}

but then other components expected for successful operation (e.g. {blah}-controller) will not show up.

i have found that the following setting will remediate this issue:

controller:
  admissionWebhooks:
    patch:
      podAnnotations:
        # https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/#policy
        #
        sidecar.istio.io/inject: "false"

this may or may not qualify as a bug, but i'm leaving this here in case it helps anyone else 👍

[leon0306]

i'm using ingree-nginx v0.41.2. after disable webhook function still can't injection istio sidecar into ingress-nginx-controller

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[natehouk]

This seems like a bug to me. Can we reopen this? Why was it closed?

[Antvirf]

Ran into this and did a bit of digging, this issue is about Kubernetes, Job containers and Sidecars, it isn't specific to ingress-nginx. Looks like it may be addressed in this issue, milestoned for release with kubernetes 1.28.

TLDR; Any job started in a namespace where sidecar injection is enabled gets a sidecar, but this sidecar pod does not know if/when the original job pod completed. The job that is set up when installing ingress-nginx works fine and completes, but the Istio sidecar pod runs till infinity, leaving the job in incomplete state.

Solution 1: Disable sidecar injection

Exactly as suggested by the issue author, add this to your values.yaml:

controller:
  admissionWebhooks:
    patch:
      podAnnotations:
        sidecar.istio.io/inject: "false"

Solution 2: Manually send the stop command to the job container

As per this post for the curl command that you need to send to istio-proxy, here's a couple one-liners for executing the call in the relevant container, note example namespace as EXAMPLE:

# admission-create
kubectl exec -it $(kubectl get pods -o custom-columns=":metadata.name" -A | grep nginx-admission-create) -n EXAMPLE -c istio-proxy -- curl -sf -XPOST http://127.0.0.1:15020/quitquitquit

# admission-patch
kubectl exec -it $(kubectl get pods -o custom-columns=":metadata.name" -A | grep nginx-admission-create) -n EXAMPLE -c istio-proxy -- curl -sf -XPOST http://127.0.0.1:15020/quitquitquit

References:

• Main issue in Kubernetes where the work is being done
• Relevant issue in Kubernetes
• Primary issue in the Istio repo