CPU and Memory stats not displayed for pods in Kubernetes

[dhop90]

Description

Currently a homepage service does not display cpu or memory stats for kubernetes pods. This is both true for services defined in services.yaml and for auto service discovery using ingress annotations:

service.yaml configuration :
- SpeedTest:
icon: speedtest-tracker.png
description: Test internet upload and download speeds
href: https://speedtest..org
ping: https://speedtest..org
statusStyle: dot
namespace: speedtest
app: speedtest
podSelector: ''
widget:
type: speedtest
url: http://speedtest.speedtest.svc.cluster.local

ingress annotation configuration:
annotations:
gethomepage.dev/container: speedtest
gethomepage.dev/description: Test internet upload and download speeds
gethomepage.dev/enabled: 'true'
gethomepage.dev/group: Monitoring
gethomepage.dev/href: https://speedtest..org
gethomepage.dev/icon: speedtest-tracker.png
gethomepage.dev/name: SpeedTest
gethomepage.dev/namespace: speedtest
gethomepage.dev/app: speedtest
gethomepage.dev/ping: https://speedtest..org
gethomepage.dev/podSelector: ''
gethomepage.dev/statusStyle: 'dot'
gethomepage.dev/weight: '1'
gethomepage.dev/widget.type: speedtest
gethomepage.dev/widget.url: http://speedtest.speedtest.svc.cluster.local

left image is for the service defined in ingress annotation

• note the gethomepage.dev/statusStyle: 'dot' does not work when defined in ingress annotation (separate issue)
• no CPU or memory stats are displayed, instead it reports as being OFFLINE

right image is for the service defined in services.yaml

• note that CPU and memory display zero values

Steps to reproduce

configure service in service.yaml and kubernetes ingress per configuration -

service.yaml configuration :
- SpeedTest:
icon: speedtest-tracker.png
description: Test internet upload and download speeds
href: https://speedtest..org
ping: https://speedtest..org
statusStyle: dot
namespace: speedtest
app: speedtest
podSelector: ''
widget:
type: speedtest
url: http://speedtest.speedtest.svc.cluster.local

ingress annotation configuration:
annotations:
cert-manager.io/cluster-issuer: cert-manager-webhook-duckdns-production
gethomepage.dev/container: speedtest
gethomepage.dev/description: Test internet upload and download speeds
gethomepage.dev/enabled: 'true'
gethomepage.dev/group: Monitoring
gethomepage.dev/href: https://speedtest..org
gethomepage.dev/icon: speedtest-tracker.png
gethomepage.dev/name: SpeedTest
gethomepage.dev/namespace: speedtest
gethomepage.dev/app: speedtest
gethomepage.dev/ping: https://speedtest..org
gethomepage.dev/podSelector: ''
gethomepage.dev/statusStyle: 'dot'
gethomepage.dev/weight: '1'
gethomepage.dev/widget.type: speedtest
gethomepage.dev/widget.url: http://speedtest.speedtest.svc.cluster.local

homepage version

v0.7.0 (c18caf9, Sep 29, 2023)

Installation method

Docker

Configuration

Kubernetes cluster using docker

kubernetes.yaml 
---
# sample kubernetes config
#
mode: cluster

kubernetes deployment:
      containers:
        - name: homepage
          image: ghcr.io/benphelps/homepage:latest
          ports:
            - name: http
              containerPort: 3000
              protocol: TCP
          env:
            - name: TZ
              value: America/Chicago
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccountName: homepage
      serviceAccount: homepage
      automountServiceAccountToken: true
      securityContext: {}
      schedulerName: default-scheduler
      enableServiceLinks: true

Container Logs

no log messages outputted

Browser Logs

No response

Troubleshooting

see steps to reproduce above

Other

No response

Before submitting, I have made sure to

• Check the documentation
• Follow the troubleshooting guide (please include output above if applicable).
• Search existing issues and discussions.

[shamoon]

No one else has reported this and it hasn't changed lately. I cant reproduce it in a simple test setup.

I dont use k8s but in general I have to wonder if this is a configuration thing app / namespace / podSelector

[dhop90]

Sorry, not following what you are looking for, If you mean, look in the devTools, this is what I found:

There are calls to :
https://homepage.domain.org/api/kubernetes/status/speedtest/speedtest?podSelector=
304 Not Modified
assuming this is from the services.yaml configuration

using curl
curl -k https://homepage.domain.org/api/kubernetes/status/speedtest/speedtest?podSelector=

Found.

I'll disable authentik and see if that changes anything

https://homepage.domain.org/api/kubernetes/status/speedtest/speedtest-ingress?
404 Not Found
assuming this is from the k8s ingress annotation

[shamoon]

Yes those calls should return data

[dhop90]

disabled authentik, but no change

[shamoon]

But... what do they return?

[shamoon]

The first one, the 404 is clearly not returning anything. But the fact that it's a 404 makes me think it's a config thing. This is just homepage using the k8s API https://github.com/benphelps/homepage/blob/861ab32ca30fbe8e93f08e8fcc8a2dc06c809984/src/pages/api/kubernetes/status/%5B...service%5D.js

[dhop90]

using a browser to https://homepage.domain.org/api/kubernetes/status/speedtest/speedtest?podSelector=

returns:
{"status":"running"}

[dhop90]

https://homepage.domain.org/api/kubernetes/status/speedtest/speedtest-ingress?

returns:
{"error":"not found"}

[shamoon]

Perhaps not from the homepage container, because if it was:
https://github.com/benphelps/homepage/blob/861ab32ca30fbe8e93f08e8fcc8a2dc06c809984/src/components/services/kubernetes-status.jsx#L14

[shamoon]

Oh sorry we're crossing streams

[shamoon]

Status works for services.yaml one, in that case we need to see api/kuberbetes/stats

Status fails for ingress one, 404 tells me its config'd wrong

[dhop90]

I agree. Lets concentrate on the services.yaml config for now.

/api/kubernetes/stats/speedtest/speedtest?podSelector=

returns:
{"stats":{"mem":0,"cpu":0,"cpuLimit":1,"memLimit":512000000,"cpuUsage":0,"memUsage":0}}

[shamoon]

Hmm, well there is some simple math being done but basically those metrics are coming from https://github.com/benphelps/homepage/blob/861ab32ca30fbe8e93f08e8fcc8a2dc06c809984/src/pages/api/kubernetes/stats/%5B...service%5D.js#L69 i.e. https://github.com/kubernetes-client/javascript

Im sure you could peel back the onion one layer more and figure out what stats your k8s install is returning. Again this hasn't changed recently so im less suspicious theres an error in the homepage logic (its just math to add things up).

Would be easy to figure out if running from source, but realize that itself may not be easy.

Is there something about k8s requiring access to metrics?

[dhop90]

access control is through the "homepage" service account which I gave cluster role binding to the homepage cluster role which I gave unlimited access to all resources, apiGroups, and verbs. So basically it has root access to everything in the cluster.

I'll try to setup a development environment so I can run from source

thanks for the help.

[dhop90]

Solved this issue. Turns out there where some mis-configuration in the cluster and with homepage.

I was seeing a couple of issues:

1. Cluster pod stats were not being displayed
   It turns out I didn't have a metrics-server deployed to the cluster. Executing "kubectl top nodes" would work but "kubectl top pods" would error. Installed metrics-server from https://github.com/kubernetes-sigs/metrics-server and everything started working again.

2. Configuration issues with services.yaml
   The most import elements for connecting to a pod are "namespace, app, and podSelector". If there is only a single pod in the namespace then setting podSelector to '' is sufficient. This will return a combined total of all pods in the name space. If you have multiple pods and want to get the stats for a specific pod, then you'll need to set podSelector to match a label. For example, I have a montoring namespace with multiple pods (prometheus, grafana, and alertmanager).

The config for these services is:
- Prometheus:
description: Self-hosted systems monitoring and alerting toolkit
icon: prometheus.png
href: https://prometheus.domain.org
ping: https://prometheus.domain.org
statusStyle: dot
namespace: monitoring
app: prometheus
podSelector: app.kubernetes.io/name=prometheus-k8s-0
widget:
type: prometheus
url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
- Alertmanager:
description: Self-hosted application that handles alerts sent by client applications
icon: prometheus.png
href: https://alertmanager.domain.org
ping: https://alertmanager.domain.org
statusStyle: dot
namespace: monitoring
app: alertmanager
podSelector: app=alertmanager
- Grafana:
description: Self-hosted open source analytics & observability platform
icon: grafana.png
href: https://grafana.domain.org
ping: https://grafana.domain.org
statusStyle: dot
namespace: monitoring
app: grafana
podSelector: app=grafana
widget:
type: grafana
url: http://grafana.monitoring.svc.cluster.local
key:

3. Configuration issues with Ingress and potential bug(s)
   It seems that when constructing the api call to the cluster the ingress name is used and not the app name. For example I have an ingress named "speedtest-ingress" for the ingress app. When the api call gets constructed it tries to call /api/kubernetes/stats/speedtest/speedtest-ingress. If I rename the ingress to match the name of the app, i.e rename speedtest-ingress to speedtest the proper api is constructed and the api call is successful.

The other potential bud I found is that the "gethomepage.dev/statusStyle: dot" is ignored.

Here is a working config (except for statusStyle):
annotations:
gethomepage.dev/description: Adminer
gethomepage.dev/enabled: 'true'
gethomepage.dev/group: Monitoring
gethomepage.dev/icon: adminer.png
gethomepage.dev/name: adminer
gethomepage.dev/app: adminer
gethomepage.dev/namespace: adminer
gethomepage.dev/podSelector: 'io.kompose.service=adminer'
gethomepage.dev/weight: '1'
gethomepage.dev/ping: https://adminer.domain.org
gethomepage.dev/statusStyle: dot

podSelector must match a label in the pod

[shamoon]

The other potential bud I found is that the "gethomepage.dev/statusStyle: dot" is ignored.

This has been fixed already.

Glad it's sorted, and yea. Kubernetes seems complicated AF...

[dhop90]

what about the using the ingress name instead of app name in constructing the api call? Yes I agree k8s is complicated AF.

[shamoon]

Im not inclined to change things like that, I'd have to dig back into the weeds of it (not my initial implementation anyway). My guess is after this being available for almost a year or so and no major complaints, it 'works' as-is for most people...

[github-actions[bot]]

This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns.