Make IAP config robust to updating the Ingress

[jlewi]

See #545 for examples of how certain changes to the config (e.g. ingress) can cause the loadbalancer to end up in a misconfigured state e.g.

• Loadbalancer uses the wrong path for the health check
• Timeout on the backend service reverts to the default of 30 seconds
• IAP gets disabled

We'd like to find a way to properly detect this and run the corrective steps.
The script used in the envoy container as an init script has most of the commands needed to correct the setup; we just need to figure out how to trigger it when things break.

There are some ideas in #545.

[danisla]

Any change in the nodePort or backend service ID breaks IAP.

It looks like the service nodePort changes every time you run ks apply ${ENV} -c iap-ingress. I would actually expect this to not modify the service nodePort if if the service already exists but for some reason it's being updated.

#552 is a working fix that converts the initContainer into a sidecar. It periodically verifies that the backend service ID has not changed. If it does change, it exits with status 1 causing K8S to restart the container and reconfigure IAP. It also restarts envoy to pull the new config.

[danisla]

ksonnet/ksonnet#200 is related to the issue regarding the nodeport changing whenever ks apply is run.

[jlewi]

Thanks

/assign @danisla

[jlewi]

@danisla looks like #552 has been committed. Should we close this issue or is there more work to be done to make it robust?