metrics-server error "Client.Timeout exceeded while awaiting headers"

[snivou]

I have the following error when I view api status after deploying metrics-server in k8s v1.10.2.

[root@master-1 metrics-server]# kubectl get apiservices v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","kubernetes.io/cluster-service":"true"},"name":"v1beta1.metrics.k8s.io","namespace":""},"spec":{"group":"metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"metrics-server","namespace":"kube-system"},"version":"v1beta1","versionPriority":100}}
creationTimestamp: 2018-07-16T07:40:09Z
labels:
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/cluster-service: "true"
name: v1beta1.metrics.k8s.io
resourceVersion: "591204"
selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
uid: 7740f0f3-88cb-11e8-afbb-02e387bb013c
spec:
caBundle: null
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
status:
conditions:

• lastTransitionTime: 2018-07-16T07:40:09Z
  message: 'no response from https://10.10.84.147:443: Get https://10.10.84.147:443:
  net/http: request canceled while waiting for connection (Client.Timeout exceeded
  while awaiting headers)'
  reason: FailedDiscoveryCheck
  status: "False"
  type: Available

/kind bug
/sig api-machinery

[sujithvs-dev]

I am also getting same error with kubernetes v1.11.0 version.

[hzxuzhonghu]

/assign

[hzxuzhonghu]

I have tested with v1.11.0-alpha.0 and there is no such issue. I think you may have set https_proxy in your env ot your cluster ip is not accessble.

One way to check is curl https://10.10.84.147:443 by hand.

[snivou]

I did not set https_prxoy. I used a binary deployment. The master node only installs apiserver, controller-manager, scheduler and is only responsible for managing the cluster. The node node is a kubelet or a proxy.

[hzxuzhonghu]

Can you access https://10.10.84.147:443 by hand? Is the container network ok?

[snivou]

Yes, and other addons are completely normal

[hzxuzhonghu]

The master node only installs apiserver, controller-manager, scheduler and is only responsible for managing the cluster.

I do not find kube-proxy.

Can you access from master ?

[hzxuzhonghu]

/remove sig-apimachinery

[sujithvs-dev]

@hzxuzhonghu,

I am also getting same error,
Message: no response from https://172.22.152.153:443: Get https://172.22.152.153:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I connected to container where metrics-server runs and there also I am not able to telnet 443 port. But I can see 443 is in listening state.

How can I troubleshoot connectivity between APIservice and metric-service. I am not able to connect metric-server service from apiserver.

Thanks,
Sujith

[hzxuzhonghu]

@sujithvs74
First: make sure kube-proxy is up on your master.

Second: make sure apiserver can access metric-server service. If not, it seems that your container network is not ok. Check whether your cni plugin is well installed.

[sujithvs-dev]

@hzxuzhonghu ,

First: kupe-proxy server is running fine on my master.
Second : I am not seeing any issue with container network as of now. One other thing, I am not able to connect to metrics-service from container itself.
telnet localhost 443. Is this expected output?

[hzxuzhonghu]

@sujithvs74 Absolutely not

[fedebongio]

/remove-sig api-machinery

[k8s-ci-robot]

@DevOps-Alvin: There are no sig labels on this issue. Please add a sig label.

A sig label can be added by either:

1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
   e.g., @kubernetes/sig-contributor-experience-<group-suffix> to notify the contributor experience sig, OR

2. specifying the label manually: /sig <group-name>
   e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. See the group list.
The <group-suffix> in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hzxuzhonghu]

/unassign

[snivou]

@hzxuzhonghu The problem has been solved. thank you very much

[cx1981]

@hwdevops I am also getting same error with kubernetes v1.12.0 version. How is this problem solved?

[ChrisCooney]

Experiencing this exact issue too.

[benjamin-maynard]

@ChrisCooney did you resolve?

[snivou]

@cx1981
Make sure that the master node installs kube-proxy and network components (eg: calico). Referring to the implementation of kubeadm, you only need to configure the master node as node, and then configure Taint to disable scheduling.

[ChrisCooney]

@ChrisCooney did you resolve?

We found that a version of heapster that was installed in a different namespace was preventing our metrics server from working. We didn't diagnose exactly why though.

[lolspider]

running flannel service on master nodes resolve my problem (Client.Timeout exceeded while awaiting headers)

[sukai127]

Get it work by running master as node, (run kubelet, kube-proxy, calico on master nodes).

The reason may be apiservice object need lookup metric-server svc to get metric.

You can run kubectl describe apiservice v1beta1.metrics.k8s.io get more details.

the result should be :

API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2018-12-19T08:54:57Z
  Resource Version:    2923
  Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  UID:                 c30804bb-036b-11e9-8b9a-fa163e53032f
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            metrics-server
    Namespace:       kube-system
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2018-12-19T08:55:02Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

[autumn1023]

thank you @hzxuzhonghu mycluster one node is show "Unable to connect to the server: Bad request"
when I set http_proxy in /etc/profile

[RamanPndy]

@hzxuzhonghu The problem has been solved. thank you very much

@stupidlittlemonkey How did you solve this problem ? I'm facing the same problem and used flannel as network plugin and kube-proxy is also throwing some errors showing it's not able to list services and nodes. I'm trying to install metric-server service.

Please suggest me any possible solution of this.

[2018-11-27]

在k8s v1.10.2中部署metrics-server后查看api状态时出现以下错误。

[root @ master-1metrics-server]＃kubectl获取apiservices v1beta1.metrics.k8s.io -o yaml
apiVersion：apiregistration.k8s.io/v1
类型：APIService
元数据：
注释：
kubectl.kubernetes.io/last-applied-配置：
{“ apiVersion”：“ apiregistration.k8s.io/v1beta1”，“种类”：“ APIService”，“元数据”：{“注释”：{}，“标签”：{“ addonmanager.kubernetes.io/mode”： “ Reconcile”，“ kubernetes.io/cluster-service":"true"},"name":"v1beta1.metrics.k8s.io","namespace":""},"spec":{"group”： “ metrics.k8s.io”，“ groupPriorityMinimum”：100，“ insecureSkipTLSVerify”：true，“ service”：{“ name”：

addonmanager.kubernetes.io/mode：协调
kubernetes.io/cluster-service：“true”
名称：v1beta1.metrics.k8s.io
resourceVersion：“ 591204”
selfLink：/apis/apiregistration.k8s.io/v1/apiservices/v1beta1 .metrics.k8s.io
uid：7740f0f3-3-cc-11e8-afbb-02e387bb013c
规格：
caBundle：空
组：metrics.k8s.io
groupPriorityMinimum：100
insecureSkipTLSVerify：true
服务：
名称：metrics-server
名称空间：kube-system
版本：v1beta1
版本优先级：100
状态：
条件：

• lastTransitionTime：2018-07-16T07：40：09Z
  消息：“没有从响应https://10.10.84.147:443：GET https://10.10.84.147:443：
  净/ HTTP：请求在等待连接取消（客户
  等待标头时超出了超时时间。”
  原因：FailedDiscoveryCheck
  状态：“假”
  类型：可用

/种类错误
/ sig api-machinery

How do you solve this problem? I am facing the same problem as your message. I can't sleep at night for this. But I can get indicators by manually executing curl - K {metrics server: Port}. It seems that apiservice cannot communicate with metrics server.

[YangShuting]

I met the same problem and now it's been solved.
1、When I entered：
kubectl get --raw "http://172.21.0.8:6443/apis/metrics.k8s.io/v1beta1/nodes" | jq .
appeared：
Error from server (ServiceUnavailable): the server is currently unable to handle the request
Typed：
kubectl describe apiservice v1beta1.metrics.k8s.io
and the 'Message' showed：
no response from https://10.254.22.174:443: Get https://10.254.22.174:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Then I logged the metrics-server pod：
kubectl logs -f metrics-server-699dcc97bf-qnxcj
and it showed me ：
unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:172.21.0.12: unable to get CPU for container "kube-state-metrics" in pod monitoring/kube-state-metrics-59bf5b787c-6m5hl on node "172.21.0.12", discarding data: missing cpu usage metric
I guess the metrics-server could not connect with the apiserver, so I checked the kube-apiserver exec args, when I added below args, finally it worked.
--proxy-client-cert-file=/etc/kubernetes/ssl/metrics-proxy.pem \ --proxy-client-key-file=/etc/kubernetes/ssl/metrics-proxy-key.pem \ --requestheader-allowed-names=aggregator \ --requestheader-group-headers=X-Remote-Group \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-username-headers=X-Remote-User \

kubectl version: v1.14.0