Kube-Prometheus can't access k3s kubernetes version

[mehdibenfeguir]

Environmental Info:
K3s Version:

k3s version v1.30.0+k3s1 (1454953)
go version go1.22.2
Node(s) CPU architecture, OS, and Version:

Linux rageeb-onprem 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:

NAME STATUS ROLES AGE VERSION
rageeb-onprem Ready control-plane,master 5d12h v1.30.0+k3s1
Describe the bug:

I installed the kube-prometheus stack latest version with these values

prometheusOperator:
tls:
enabled: false
adminissionWebhooks:
enabled: false

I'm getting this error on the opeator
k logs kube-prometheus-kube-prome-operator-5554cdff77-qhfz6 -f

level=info ts=2024-05-22T10:33:57.140735311Z caller=main.go:186 msg="Starting Prometheus Operator" version="(version=0.73.2, branch=refs/tags/v0.73.2, revision=20beb0b00b0bb61840ec2461ace4f22e4037b474)"
level=info ts=2024-05-22T10:33:57.140786212Z caller=main.go:187 build_context="(go=go1.22.2, platform=linux/amd64, user=Action-Run-ID-8756162704, date=20240419-15:41:23, tags=unknown)"
level=info ts=2024-05-22T10:33:57.140811012Z caller=main.go:198 msg="namespaces filtering configuration " config="{allow_list="",deny_list="",prometheus_allow_list="",alertmanager_allow_list="",alertmanagerconfig_allow_list="",thanosruler_allow_list=""}"

level=error ts=2024-05-22T10:34:27.142774894Z caller=main.go:222 msg="failed to request Kubernetes server version" err="Get "https://10.43.0.1:443/version\": dial tcp 10.43.0.1:443: i/o timeout"
Steps To Reproduce:

• Installed K3s:

Expected behavior:

Actual behavior:

Additional context / logs:

[brandond]

I don't know where this pod is running, but for some reason it's unable to access the in-cluster apiserver endpoint. Normally this is handled by kube-proxy and the CNI. Verify that you've opened the correct CNI ports between your agent and server nodes: https://docs.k3s.io/installation/requirements#inbound-rules-for-k3s-nodes