Cannot install Gloo Edge with validation webhook set to Fail

[rinormaloku]

Gloo Edge Version

1.10.x (latest stable)

Kubernetes Version

No response

Describe the bug

Cannot install Gloo Edge when the validation webhook failure policy is set to Fail

Steps to reproduce the bug

Execute.

helm install gloo glooe/gloo-ee --namespace gloo-system --version 1.10.15 --set-string license_key=$LICENSE_KEY_EDGE -f - <<EOF
gloo:
  gateway:
    validation:
      failurePolicy: Fail
EOF

Expected Behavior

Installation works.

Additional Context

**Actual behavior **: Installation fails with the following error:

Error: INSTALLATION FAILED: Internal error occurred: failed calling webhook "gateway.gloo-system.svc": Post "https://gateway.gloo-system.svc:443/validation?timeout=10s": dial tcp 10.96.151.246:443: connect: connection refused

Under the logs we find out why:

$ kubectl logs deploy/gateway                         

{"level":"info","ts":"2022-04-01T09:49:51.698Z","logger":"gateway.v1.event_loop","caller":"v1/setup_event_loop.sk.go:57","msg":"event loop started","version":"1.10.17"}
{"level":"fatal","ts":"2022-04-01T09:49:51.799Z","logger":"gateway","caller":"setuputils/main_setup.go:73","msg":"error in setup: finding bootstrap configuration: list did not find settings gloo-system.default","version":"1.10.17","stacktrace":"github.com/solo-io/gloo/pkg/utils/setuputils.Main\n\t/workspace/gloo/pkg/utils/setuputils/main_setup.go:73\ngithub.com/solo-io/gloo/projects/gateway/pkg/setup.Main\n\t/workspace/gloo/projects/gateway/pkg/setup/setup.go:13\nmain.main\n\t/workspace/gloo/projects/gateway/cmd/main.go:11\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"}

Key here is: list did not find settings gloo-system.default

So this is a chicken and egg problem, Settings has to be removed from the validation webhook.

[kevin-shelaga]

Another customer with this issue as well

[jmunozro]

Found this one in a prospect recently

[kevin-shelaga]

This might be fixed by this #4468

[soloio-bot]

Zendesk ticket #701 has been linked to this issue.

[totallyGreg]

Customer has seen this issue and noted:
To reproduce the issue,

• Set the ValidatingWebhookConfiguration's failurePolicy to Fail.

• Upgrade the Gloo Edge to a new version. During the upgrade, Helm tries to patch Upstreams extauth and rate-limit. If the Gateway pod is not upgraded yet, upgrade fails with following error:
  failed calling webhook "gateway.gloo-system.svc": Post "https://gateway.gloo-system.svc:443/validation?timeout=10s": dial tcp 192.169.18.151:443: connect: connection refused

Please note the upgrade may work if the gateway pod upgrade is completed quickly in time prior to patching the upstream resources.

[totallyGreg]

The current recommendation when using strict validation is a two step method as described in the PR #6017 that fixed issue #4468
Where the upgrade process is
1.  helm install or upgrade with values gatewaySettings.enabled: false ​

cat << EOF > values.yaml
gloo:
  gateway:
    validation:
      allowWarnings: false
      alwaysAcceptResources: false
      failurePolicy: Fail
gatewayProxies:
  gatewayProxy:
    gatewaySettings:
      enabled: false
EOF

2. Then run the helm install or upgrade with values gatewaySettings.enabled: true

[jenshu]

fix will be available in GlooEE v1.11.9 and v1.12.0-beta6