Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metrics-server start failed. status is CrashLoopBackOff #76575

Open
chicago7 opened this Issue Apr 15, 2019 · 6 comments

Comments

Projects
None yet
6 participants
@chicago7
Copy link

chicago7 commented Apr 15, 2019

What happened:
I0411 03:09:30.784560 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I0411 03:09:30.784637 1 heapster.go:72] Metrics Server version v0.2.1
I0411 03:09:30.784846 1 configs.go:61] Using Kubernetes client with master "https://10.0.0.1:443" and version
I0411 03:09:30.784862 1 configs.go:62] Using kubelet port 10255
I0411 03:09:30.785670 1 heapster.go:128] Starting with Metric Sink
I0411 03:09:31.372204 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
F0411 03:09:31.904501 1 heapster.go:97] Could not create the API server: cluster doesn't provide requestheader-client-ca-file

I try to use the below link that fix the issue, but didn't work

Azure/acs-engine#2421
Azure/acs-engine#2665
Azure/acs-engine#2264

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
aks engine v0.34.0
k8s01.json
{
"apiVersion": "vlabs",
"location": "chinaeast2",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorVersion": "1.14.1",
"kubernetesConfig": {
"clusterSubnet": "10.244.0.0/16",
"networkPlugin": "kubenet",
"networkPolicy": "calico",
"enableRbac": false,
"addons": [
{
"name": "tiller",
"enabled": false
},
{
"name": "kubernetes-dashboard",
"enabled": false
},
{
"name": "blobfuse-flexvolume",
"enabled": false
},
{
"name": "keyvault-flexvolume",
"enabled": false
},
{
"name": "rescheduler",
"enabled": true
}
],
"apiServerConfig": {
"--enable-admission-plugins": "DenyEscalatingExec,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,ExtendedResourceToleration"
}
}
},
"masterProfile": {
"count": 1,
"dnsPrefix": "test03",
"vmSize": "Standard_A2m_v2",
"firstConsecutiveStaticIP": "172.16.0.200",
"vnetSubnetId": "",
"storageProfile": "ManagedDisks"
},
"agentPoolProfiles": [
{
"availabilityProfile": "AvailabilitySet",
"name": "test02",
"count": 1,
"vmSize": "Standard_A2m_v2",
"vnetSubnetId": "",
"storageProfile": "ManagedDisks",
"osType": "Linux"
}
],
"linuxProfile": {
"adminUsername": "test02",
"ssh": {
"publicKeys": [
{
"keyData": ""
}
]
}
},
"servicePrincipalProfile": {
"clientId": "",
"secret": "*******"
}
}
}
Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): 1.14.1
  • Cloud provider or hardware configuration: Azure China
  • OS (e.g: cat /etc/os-release): ubuntu 16.04-LTS
  • Kernel (e.g. uname -a):4.15.0-1025-azure #26~16.04.1-Ubuntu SMP Tue Sep 25 11:09:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Others:
@Duchongming

This comment has been minimized.

Copy link

Duchongming commented Apr 15, 2019

查看一下证书

@neolit123

This comment has been minimized.

Copy link
Member

neolit123 commented Apr 15, 2019

/sig azure
/sig instrumentation

@andyzhangx

This comment has been minimized.

Copy link
Member

andyzhangx commented Apr 17, 2019

I tried by aks engine v0.34.0 on chinaeast2, metric server works well:

kubectl logs metrics-server-74d64b844c-gnjzg -n kube-system
I0417 02:28:34.825300       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I0417 02:28:34.825344       1 heapster.go:72] Metrics Server version v0.2.1
I0417 02:28:34.825561       1 configs.go:61] Using Kubernetes client with master "https://10.0.0.1:443" and version
I0417 02:28:34.825576       1 configs.go:62] Using kubelet port 10255
I0417 02:28:34.826458       1 heapster.go:128] Starting with Metric Sink
I0417 02:28:35.079889       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0417 02:28:35.218543       1 heapster.go:101] Starting Heapster API server...
[restful] 2019/04/17 02:28:35 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2019/04/17 02:28:35 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0417 02:28:35.219484       1 serve.go:85] Serving securely on 0.0.0.0:443

@chicago7 the only difference is I set network plugin and policy as following:

    "networkPlugin": {
      "value": "azure"
    },
    "networkPolicy": {
      "value": ""
    },

I am not sure calico works on azure china now.

@chicago7

This comment has been minimized.

Copy link
Author

chicago7 commented Apr 17, 2019

@ andyzhangx thanks for your reply, i will try it

@chicago7

This comment has been minimized.

Copy link
Author

chicago7 commented Apr 17, 2019

@andyzhangx i change the network plugin and policy value. but the issue still happened.

NAME READY STATUS RESTARTS AGE
azure-cni-networkmonitor-2272p 1/1 Running 0 3m
azure-cni-networkmonitor-4bt9n 1/1 Running 0 3m
azure-ip-masq-agent-ftwzn 1/1 Running 0 3m
azure-ip-masq-agent-kktnd 1/1 Running 0 3m
coredns-5dc99c87cb-r8vw5 1/1 Running 0 3m
kube-addon-manager-k8s-master-15989644-0 1/1 Running 0 2m
kube-apiserver-k8s-master-15989644-0 1/1 Running 0 2m
kube-controller-manager-k8s-master-15989644-0 1/1 Running 0 2m
kube-proxy-mtmws 1/1 Running 0 3m
kube-proxy-nxxfr 1/1 Running 0 3m
kube-scheduler-k8s-master-15989644-0 1/1 Running 0 2m
metrics-server-74d64b844c-l4tkc 0/1 CrashLoopBackOff 4 3m
rescheduler-7b7b488f66-ffhr6 1/1 Running 0 3m

I0417 09:36:34.973692 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I0417 09:36:34.973761 1 heapster.go:72] Metrics Server version v0.2.1
I0417 09:36:34.973897 1 configs.go:61] Using Kubernetes client with master "https://10.0.0.1:443" and version
I0417 09:36:34.973905 1 configs.go:62] Using kubelet port 10255
I0417 09:36:34.974624 1 heapster.go:128] Starting with Metric Sink
I0417 09:36:35.949188 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
F0417 09:36:36.253041 1 heapster.go:97] Could not create the API server: cluster doesn't provide requestheader-client-ca-file

@AhmedMoAli

This comment has been minimized.

Copy link

AhmedMoAli commented Apr 18, 2019

Hello,

Any feedback about this ticket?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.