Deploying Metrics to AKS configuration

[jvandervelden]

When deploying the metrics pods via lens to an AKS cluster the pods can't find nodes to run on.

Looking at it the pods have this node selector on them: kubernetes.io/os: linux

While AKS nodes are linux, they by default don't have this label on them causing the pod failure.

Cluster Type: Azure Kubernetes System (AKS)
Nodes: Linux aks-agentpool-XXXX 4.15.0-1060-azure #65-Ubuntu SMP Wed Sep 18 08:55:51 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Client OS: Windows 10

Error Events:

Event: prometheus-0.15ca241e30b1e6e7
Message 0/8 nodes are available: 8 node(s) didn't match node selector.
Created 1h 5m ago (2019-10-03T12:52:12Z)
Namespace lens-metrics
Reason FailedScheduling
Source default-scheduler
First seen 1h ago 2019-10-03T12:52:12Z
Last seen <1m ago 2019-10-03T13:57:15Z
Count 2931
Type Warning
Involved object Pod prometheus-0

Event: kube-state-metrics-54558ff88b-rfv2z.15ca241dac39ce92
Message 0/8 nodes are available: 8 node(s) didn't match node selector.
Created 1h 6m ago (2019-10-03T12:52:09Z)
Namespace lens-metrics
Reason FailedScheduling
Source default-scheduler
First seen 1h ago 2019-10-03T12:52:09Z
Last seen 1m ago 2019-10-03T13:57:14Z
Count 2931
Type Warning
Involved object Pod kube-state-metrics-54558ff88b-rfv2z

Once I removed the node selector from both the Prometheus statefulset and the Metrics deployment it works fine.

[jnummelin]

@jvandervelden Which version of AKS are you running? All my tests with AKS have set the beta.kubernetes.io/os: linux labels on the nodes. 🤔

Very surprising if the nodes do not set that as it is pretty "standard" node label set by kubelet itself.

[jvandervelden]

You're right I looked again and they do have the beta.kubernetes.io/os: linux label.

[jnummelin]

So the problem is that kubernetes.io/os: linux is only available on some envs and in some there's beta.kubernetes.io/os: linux. We will tackle that in next release. (fix PR already merged)

[jnummelin]

This is fixed in 2.1.0 release.