Presubmits failing for basic auth timeout waiting for backend-updater to be ready

[jlewi]

Example PR:
kubeflow/kubeflow#4275 (comment)

[jlewi]

Looking at the events for the backend-updater-pod it is crashing.

Logs for the pod show

Error unable to fetch PROJECT from compute metadata

Why is this only basic auth and not IAP?

Is this happening in postsubmits?

[jlewi]

On master it looks like the recent run of periodic succeeded
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/kubeflow-periodic-master/1182776231863521280

[jlewi]

Its possible this is related to the break in GKE with workload identity.
#48

Although I don't know why it would only be showing up in basic_auth tests and not IAP.

[jlewi]

It looks ike the periodic master runs our stale.
Last run is from 10-11 18:57 pdt
https://k8s-testgrid.appspot.com/sig-big-data#kubeflow-periodic-master

[jlewi]

Looks like this is fixed by kubeflow/manifests#498 which pins to 1.14.6-gke.2