Error: couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request

[jlewi]

/kind bug

What steps did you take and what happened:

Spurious error message from kfctl when deploying on GCP

Kfctl prints the following error out

E1025 15:22:05.967555   43645 memcache.go:135] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request

This error appears to be harmless but it will confuse users so we should figure out where its coming from and suppress it.

[issue-label-bot]

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.99. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[schmidt-jake]

I'm encountering the same issue.

[jlewi]

A quick mitigation would be to have apply print out a success message when its done so that users know the deployment succeeded.

cc @lluunn

[jlewi]

#4443 onto release branch.

[jlewi]

Using basic auth on GCP and

kubeflow/kbueflow - v0.7.0-rc.7-3-g8dbde9d8
kubeflow/manifests - v0.7.0-rc.2-19-g317c0d8

I see the success message added by #4443 but I don't see the custom metrics error.

INFO[0568] Downloading secret user-gcp-sa from namespace kubeflow  filename="gcp/gcp.go:1835"
INFO[0568] Creating secret user-gcp-sa to namespace kubeflow-jlewi  filename="gcp/gcp.go:1840"
INFO[0568] Generating PodDefault in namespace kubeflow-jlewi; APIVersion kubeflow.org/v1alpha1  filename="gcp/gcp.go:1770"
INFO[0575] Applied the configuration Successfully!       filename="cmd/apply.go:72"

[jlewi]

With IAP

E1104 05:56:10.033789  242816 memcache.go:135] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
INFO[0639] Applied the configuration Successfully!       filename="cmd/apply.go:72"

So we still have the error about custom metrics but we have the success metric after it.

Downgrading to P2 because this looks like a fit and finish issue now to figure out what to do about the spurious error message.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.