dev.kubeflow.org 502s

[jlewi]

I'm getting 502s on deve.kubeflow.org

I believe @ankushagarwal just redeployed the latest Kubeflow.

Following iap.md

• Backend is reported as unhealthy
• Health check is reported as "/" not "/healthz"
• IAP is disabled
• Timeout is set to 30 seconds.

My conjecture is when we redeployed all the backend settings got reset and we didn't reinitialize

/cc @danisla @ankushagarwal

[jlewi]

• Used the UI to change the health check to /healthz
• Took some time but eventually health backends reported as working

Now I get

Required JWT token is missing

Because IAP is disabled.

[danisla]

The initContainer on the envoy pods are what re-enable IAP. Deleting the envoy deployment and running ks apply again should reconfigure it.

[jlewi]

@danisla Thanks just figured that out.

It looks like when @ankushagarwal ks apply didn't cause the envoy containers to get restarted.

[danisla]

If the ingress or service spec is what changed, we might be able to add a coupling in the jsonnet spec that rolls the pods whenever the ingress or service changes. This is done in Helm with an annotation and a checksum.

[jlewi]

Would it make sense to turn the script into an agent that periodically runs a bunch of checks?

This would be more work I think.

[jlewi]

Its working now that I kicked one of the envoy pods.

[danisla]

If the liveness probe on the pod was somehow coupled to the JWT_AUDIENCE being correct, then k8s would automatically restart the pods when it changed and self-correct.

[jlewi]

Lets use #550 to figure out a better fix.