Disabling config validation doesn't actually seem to disable it

[Stono]

Bug description
Hey,
We have .Values.global.configValidation: false in our installation options, however we still get a ValidatingWebhookConfiguration generated from istioctl generate-manifests.

We explicitly ignore that file in our templating to work around this, but then istiod is full of spammy logs like this:

istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:43.926055Z    info    validationController    Reconcile(enter): retry dry-run creation of invalid config
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:43.928898Z    info    validationController    Not ready to switch validation to fail-closed: dummy invalid config not rejected
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:43.929126Z    warn    validationController    validatingwebhookconfiguration.admissionregistration.k8s.io "istiod-istio-system" not found
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:44.929083Z    info    validationController    Reconcile(enter): retry dry-run creation of invalid config
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:44.978814Z    info    validationController    Not ready to switch validation to fail-closed: dummy invalid config not rejected
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:44.979049Z    warn    validationController    validatingwebhookconfiguration.admissionregistration.k8s.io "istiod-istio-system" not found
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:45.979007Z    info    validationController    Reconcile(enter): retry dry-run creation of invalid config
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:45.982631Z    info    validationController    Not ready to switch validation to fail-closed: dummy invalid config not rejected
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:45.982835Z    warn    validationController    validatingwebhookconfiguration.admissionregistration.k8s.io "istiod-istio-system" not found
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:46.982778Z    info    validationController    Reconcile(enter): retry dry-run creation of invalid config
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:46.986500Z    info    validationController    Not ready to switch validation to fail-closed: dummy invalid config not rejected
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:46.986701Z    warn    validationController    validatingwebhookconfiguration.admissionregistration.k8s.io "istiod-istio-system" not found
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:47.986708Z    info    validationController    Reconcile(enter): retry dry-run creation of invalid config
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:47.989817Z    info    validationController    Not ready to switch validation to fail-closed: dummy invalid config not rejected
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:47.990110Z    warn    validationController    validatingwebhookconfiguration.admissionregistration.k8s.io "istiod-istio-system" not found
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:48.990059Z    info    validationController    Reconcile(enter): retry dry-run creation of invalid config
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:48.993643Z    info    validationController    Not ready to switch validation to fail-closed: dummy invalid config not rejected
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:48.993841Z    warn    validationController    validatingwebhookconfiguration.admissionregistration.k8s.io "istiod-istio-system" not found
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:49.993844Z    info    validationController    Reconcile(enter): retry dry-run creation of invalid config
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:49.997637Z    info    validationController    Not ready to switch validation to fail-closed: dummy invalid config not rejected
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:49.997842Z    warn    validationController    validatingwebhookconfiguration.admissionregistration.k8s.io "istiod-istio-system" not found

Which looks like its continually running some sort of test?

We want to minimise the amount of change between 1.5 -> 1.6, so do not wish to run with a webhook validating resources for now.

[ ] Docs
[x] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[x] User Experience
[x] Developer Infrastructure

Expected behavior
The config option to disable the webhook and any logic in istiod around config validation.

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)
1.6.12

How was Istio installed?
Helm

Environment where bug was observed (cloud vendor, OS, etc)

[esnible]

I can see that on a fresh install this setting suppresses the webhook

diff <(istioctl-1.7.3 manifest generate --set values.global.configValidation=false) <(istioctl-1.7.3 manifest generate)

I did a fresh install and saw that the setting is respected.
I installed without the setting and then reinstalled with --set values.global.configValidation=false and verified that if Istio was installed with the validating webhook, that re-installing without it doesn't delete the existing webhook.

[Stono]

@esnible perhaps this is a 1.6 bug then?

❯ ./src/1.6.12/bin/istioctl manifest generate --set values.global.configValidation=false | grep 'ValidatingWebhookConfiguration'
kind: ValidatingWebhookConfiguration

As you can see; the ValidatingWebhookConfiguration is still getting created.

And then like i mentioned in the OP, even if we remove it from our chart manually and fresh deploy 1.6 without it, istiod still spams the log level 1s attempting to set it up (but obviously won't work, as we haven't deployed ValidatingWebhookConfiguration). So it seems istiod isn't honouring that setting either.

istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:49.993844Z    info    validationController    Reconcile(enter): retry dry-run creation of invalid config
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:49.997637Z    info    validationController    Not ready to switch validation to fail-closed: dummy invalid config not rejected
istiod-8b6c5dcc6-d2sdt discovery 2020-10-10T17:36:49.997842Z    warn    validationController    validatingwebhookconfiguration.admissionregistration.k8s.io "istiod-istio-system" not found

[esnible]

It looks like the ability to turn off validation was added recently (July 10): #25372 by @linsun

We can look at back-porting this to 1.6. Alternately, your 1.6 Istio install script could get kubectl delete ValidatingWebhookConfiguration istiod-istio-system added.

[Stono]

@esnible yeah; that's what we've done - as I said we removed it but then get log spam every 1 second as istiod is still trying to setup a webhook that isn't there.

@howardjohn gave me a work around which was to also add:

  components:
    pilot:
      k8s:
        env:
        - name: VALIDATION_WEBHOOK_CONFIG_NAME
          value: ''

The combination of these two actions seems to effectively have disabled config validation in 1.6.

With regards to back porting, I think that's up to you... From a user perspective, I set configValidation which is a valid config option (see https://istio.io/v1.6/docs/ops/common-problems/validation) however it doesn't work.

I would suggest either:

1. Backporting to 1.6 or
2. Removing the config option from 1.6

To prevent user confusion

[howardjohn]

#26531 (log spam) and #25372 (remove webhook if flag is set) is what is missing. Both are in 1.7 but not 1.6