apiservice Error reporting：failing or missing response from https://10.244.2.96:6443/apis/custom.metrics.k8s.io/v1beta1: bad status from https://10.244.2.96:6443/apis/custom.metrics.k8s.io/v1beta1: 404

[jibingjie2]

What happened?:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" An error has occurred：Error from server (ServiceUnavailable): the server is currently unable to handle the request
The most direct information，apiservice Error reporting：failing or missing response from https://10.244.2.96:6443/apis/custom.metrics.k8s.io/v1beta1: bad status from https://10.244.2.96:6443/apis/custom.metrics.k8s.io/v1beta1: 404

What did you expect to happen?:
return custom.metrics.k8s.io/v1beta1 for api info
Please provide the prometheus-adapter config:

prometheus-adapter config

I use the default code without making any changes

Please provide the prometheus-adapter logs with -v=6 around the time the issue happened:

prometheus-adapter logs

I1213 00:21:06.715145 24371 loader.go:375] Config loaded from file: /root/.kube/config
I1213 00:21:06.782155 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/networking.k8s.io/v1?timeout=32s 200 OK in 33 milliseconds
I1213 00:21:06.782530 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/batch/v1beta1?timeout=32s 200 OK in 36 milliseconds
I1213 00:21:06.782716 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/authorization.k8s.io/v1?timeout=32s 200 OK in 39 milliseconds
I1213 00:21:06.783360 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/authorization.k8s.io/v1beta1?timeout=32s 200 OK in 17 milliseconds
I1213 00:21:06.815251 24371 request.go:581] Throttling request took 72.02758ms, request: GET:https://192.168.16.16:16443/apis/networking.k8s.io/v1beta1?timeout=32s
I1213 00:21:06.820741 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/objectbucket.io/v1alpha1?timeout=32s 200 OK in 34 milliseconds
I1213 00:21:06.824029 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/apiregistration.k8s.io/v1?timeout=32s 200 OK in 78 milliseconds
I1213 00:21:06.824747 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/storage.k8s.io/v1?timeout=32s 200 OK in 75 milliseconds
I1213 00:21:06.825227 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/storage.k8s.io/v1beta1?timeout=32s 200 OK in 51 milliseconds
I1213 00:21:06.825311 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/rbac.authorization.k8s.io/v1?timeout=32s 200 OK in 79 milliseconds
I1213 00:21:06.825365 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/apiregistration.k8s.io/v1beta1?timeout=32s 200 OK in 79 milliseconds
I1213 00:21:06.825581 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s 200 OK in 51 milliseconds
I1213 00:21:06.825683 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/admissionregistration.k8s.io/v1?timeout=32s 200 OK in 51 milliseconds
I1213 00:21:06.825719 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/elasticsearch.k8s.elastic.co/v1alpha1?timeout=32s 200 OK in 41 milliseconds
I1213 00:21:06.825787 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/apiextensions.k8s.io/v1beta1?timeout=32s 200 OK in 41 milliseconds
I1213 00:21:06.825840 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/scheduling.k8s.io/v1?timeout=32s 200 OK in 40 milliseconds
I1213 00:21:06.826052 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/config.istio.io/v1alpha2?timeout=32s 200 OK in 11 milliseconds
I1213 00:21:06.826100 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/kiali.io/v1alpha1?timeout=32s 200 OK in 41 milliseconds
I1213 00:21:06.826188 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/kibana.k8s.elastic.co/v1alpha1?timeout=32s 200 OK in 41 milliseconds
I1213 00:21:06.826298 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/devices.kubeedge.io/v1alpha1?timeout=32s 200 OK in 42 milliseconds
I1213 00:21:06.826348 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/logging.banzaicloud.io/v1beta1?timeout=32s 200 OK in 40 milliseconds
I1213 00:21:06.826379 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/logging-extensions.banzaicloud.io/v1alpha1?timeout=32s 200 OK in 41 milliseconds
I1213 00:21:06.826424 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/logging.banzaicloud.io/v1alpha1?timeout=32s 200 OK in 40 milliseconds
I1213 00:21:06.826483 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/apiextensions.k8s.io/v1?timeout=32s 200 OK in 42 milliseconds
I1213 00:21:06.826511 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/monitoring.kiali.io/v1alpha1?timeout=32s 200 OK in 40 milliseconds
I1213 00:21:06.826536 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/scheduling.k8s.io/v1beta1?timeout=32s 200 OK in 40 milliseconds
I1213 00:21:06.826560 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/rbac.istio.io/v1alpha1?timeout=32s 200 OK in 39 milliseconds
I1213 00:21:06.826656 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/reliablesyncs.kubeedge.io/v1alpha1?timeout=32s 200 OK in 39 milliseconds
I1213 00:21:06.826689 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/discovery.k8s.io/v1beta1?timeout=32s 200 OK in 38 milliseconds
I1213 00:21:06.826710 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/coordination.k8s.io/v1?timeout=32s 200 OK in 39 milliseconds
I1213 00:21:06.826732 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/catalog.cattle.io/v1?timeout=32s 200 OK in 37 milliseconds
I1213 00:21:06.826754 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/node.k8s.io/v1beta1?timeout=32s 200 OK in 38 milliseconds
I1213 00:21:06.826802 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/coordination.k8s.io/v1beta1?timeout=32s 200 OK in 38 milliseconds
I1213 00:21:06.826824 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/jaegertracing.io/v1?timeout=32s 200 OK in 36 milliseconds
I1213 00:21:06.826853 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/ceph.rook.io/v1?timeout=32s 200 OK in 37 milliseconds
I1213 00:21:06.826991 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/networking.k8s.io/v1beta1?timeout=32s 200 OK in 11 milliseconds
I1213 00:21:06.827018 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/management.cattle.io/v3?timeout=32s 200 OK in 12 milliseconds
I1213 00:21:06.827046 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/app.k8s.io/v1beta1?timeout=32s 200 OK in 11 milliseconds
I1213 00:21:06.830433 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/apps/v1?timeout=32s 200 OK in 87 milliseconds
I1213 00:21:06.830739 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/autoscaling/v1?timeout=32s 200 OK in 86 milliseconds
I1213 00:21:06.830805 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/events.k8s.io/v1?timeout=32s 200 OK in 86 milliseconds
I1213 00:21:06.831261 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/autoscaling/v2beta1?timeout=32s 200 OK in 86 milliseconds
I1213 00:21:06.831294 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/policy/v1beta1?timeout=32s 200 OK in 86 milliseconds
I1213 00:21:06.831320 24371 round_trippers.go:443] GET https://192.168.16.16:16443/api/v1?timeout=32s 200 OK in 86 milliseconds
I1213 00:21:06.831464 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/autoscaling/v2beta2?timeout=32s 200 OK in 86 milliseconds
I1213 00:21:06.831512 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/events.k8s.io/v1beta1?timeout=32s 200 OK in 86 milliseconds
I1213 00:21:06.831635 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/authentication.k8s.io/v1?timeout=32s 200 OK in 85 milliseconds
I1213 00:21:06.831678 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/batch/v1?timeout=32s 200 OK in 85 milliseconds
I1213 00:21:06.832224 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/metrics.k8s.io/v1beta1?timeout=32s 200 OK in 88 milliseconds
I1213 00:21:06.832461 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/extensions/v1beta1?timeout=32s 200 OK in 85 milliseconds
I1213 00:21:06.832547 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/rbac.authorization.k8s.io/v1beta1?timeout=32s 200 OK in 85 milliseconds
I1213 00:21:06.832564 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/certificates.k8s.io/v1beta1?timeout=32s 200 OK in 84 milliseconds
I1213 00:21:06.832572 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/certificates.k8s.io/v1?timeout=32s 200 OK in 85 milliseconds
I1213 00:21:06.832587 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/authentication.k8s.io/v1beta1?timeout=32s 200 OK in 85 milliseconds
I1213 00:21:06.836492 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/velero.io/v1?timeout=32s 200 OK in 4 milliseconds
I1213 00:21:06.836654 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/monitoring.coreos.com/v1?timeout=32s 200 OK in 6 milliseconds
I1213 00:21:06.839519 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/rules.kubeedge.io/v1?timeout=32s 200 OK in 8 milliseconds
I1213 00:21:06.839577 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/monitoring.coreos.com/v1alpha1?timeout=32s 200 OK in 8 milliseconds
I1213 00:21:06.839617 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/ui.cattle.io/v1?timeout=32s 200 OK in 7 milliseconds
I1213 00:21:06.839653 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/snapshot.storage.k8s.io/v1beta1?timeout=32s 200 OK in 7 milliseconds
I1213 00:21:06.839688 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/snapshot.storage.k8s.io/v1?timeout=32s 200 OK in 8 milliseconds
I1213 00:21:06.839716 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/apm.k8s.elastic.co/v1alpha1?timeout=32s 200 OK in 7 milliseconds
I1213 00:21:06.840828 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/argoproj.io/v1alpha1?timeout=32s 200 OK in 8 milliseconds
I1213 00:21:06.865702 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/custom.metrics.k8s.io/v1beta1?timeout=32s 503 Service Unavailable in 50 milliseconds
I1213 00:21:06.928155 24371 request.go:1301] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:"apiVersion,omitempty""; Kind string "json:"kind,omitempty"" }
I1213 00:21:06.957620 24371 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I1213 00:21:06.957726 24371 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I1213 00:21:06.961339 24371 round_trippers.go:443] GET https://192.168.16.16:16443/apis/custom.metrics.k8s.io/v1beta1?timeout=32s 503 Service Unavailable in 1 milliseconds
I1213 00:21:06.963644 24371 request.go:1301] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:"apiVersion,omitempty""; Kind string "json:"kind,omitempty"" }
I1213 00:21:06.963688 24371 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I1213 00:21:06.971732 24371 round_trippers.go:443] GET https://192.168.16.16:16443/api/v1/namespaces/monitoring/pods/prometheus-adapter-cb9df5dfc-cw6bb 200 OK in 3 milliseconds
I1213 00:21:06.997295 24371 round_trippers.go:443] GET https://192.168.16.16:16443/api/v1/namespaces/monitoring/pods/prometheus-adapter-cb9df5dfc-cw6bb/log?follow=true 200 OK in 17 milliseconds
I1212 14:26:27.877455 1 adapter.go:108] successfully using in-cluster auth
W1212 14:26:56.466996 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
I1212 14:26:56.568365 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1212 14:26:56.568386 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1212 14:26:56.568631 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1212 14:26:56.568657 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1212 14:26:56.568682 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1212 14:26:56.568687 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1212 14:26:56.569313 1 secure_serving.go:256] Serving securely on [::]:6443
I1212 14:26:56.569348 1 dynamic_serving_content.go:129] "Starting controller" name="serving-cert::/var/run/serving-cert/apiserver.crt::/var/run/serving-cert/apiserver.key"
I1212 14:26:56.569386 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I1212 14:26:56.669351 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1212 14:26:56.669421 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I1212 14:26:56.669625 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E1212 14:27:33.908457 1 webhook.go:224] Failed to make webhook authorizer request: Post "https://10.0.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s": context canceled
E1212 14:27:33.908701 1 errors.go:77] Post "https://10.0.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s": context canceled
E1212 14:27:33.910009 1 timeout.go:135] post-timeout activity - time-elapsed: 1.324931ms, GET "/metrics" result:
I1212 14:34:11.737538 1 trace.go:205] Trace[1980435746]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.0/tools/cache/reflector.go:167 (12-Dec-2023 14:33:55.913) (total time: 15824ms):
Trace[1980435746]: ---"Objects listed" 15824ms (14:34:11.737)
Trace[1980435746]: [15.824441381s] [15.824441381s] END
E1212 14:36:03.892035 1 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, context canceled]"
E1212 14:36:03.892661 1 webhook.go:155] Failed to make webhook authenticator request: Post "https://10.0.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": context deadline exceeded
E1212 14:36:03.896466 1 writers.go:117] apiserver was unable to write a JSON response: http: Handler timeout
E1212 14:36:03.896505 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E1212 14:36:03.897876 1 writers.go:130] apiserver was unable to write a fallback JSON response: http: Handler timeout
E1212 14:36:03.899416 1 timeout.go:135] post-timeout activity - time-elapsed: 6.961975ms, GET "/metrics" result:

kube-apiserver logs<level：error>:

E1213 00:25:43.387647 3313 available_controller.go:437] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.244.2.96:6443/apis/custom.metrics.k8s.io/v1beta1: bad status from https://10.244.2.96:6443/apis/custom.metrics.k8s.io/v1beta1: 404
E1213 00:25:43.416833 3313 crd_finalizer.go:299] elasticsearches.elasticsearch.k8s.elastic.co failed with: timed out waiting for the condition
E1213 00:25:43.699300 3313 controller.go:116] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E1213 00:25:44.030954 3313 available_controller.go:437] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.244.2.96:6443/apis/custom.metrics.k8s.io/v1beta1: bad status from https://10.244.2.96:6443/apis/custom.metrics.k8s.io/v1beta1: 404

This is most of the error messages, with a 503 sandwiched between 404

Environment:

• prometheus-adapter version: v0.9.1
• prometheus version: v2.34.0
• Kubernetes version (use kubectl version):v1.19
• operating system version:centos7.9
• Other info:
  kube-apiserve inlet：192.168.16.16：16443

[logicalhan]

/triage accepted
/assign @dgrisonnet

[shenshouer]

I met the same issue