Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nothing to do for Prometheus #1576

Closed
m-jayson opened this issue Sep 16, 2020 · 23 comments
Closed

Nothing to do for Prometheus #1576

m-jayson opened this issue Sep 16, 2020 · 23 comments

Comments

@m-jayson
Copy link

I tried enabling prometheus on my microk8s cluster on raspberry pi however it is giving me this below result.
image

It seems prometheus is not part of the default addons
image

I can access grafana
image

There is influxdb however, we like to use prometheus as we have experience on that.

I'm following this example but since there is no prometheus it looks like i cant
https://ubuntu.com/blog/monitoring-at-the-edge-with-microk8s

lastly this is my microk8s version --classic --channel=1.18/stable

@balchua
Copy link
Collaborator

balchua commented Sep 17, 2020

@mgonzaga1990 thanks for reporting the issue.
Back then like a couple of months ago, the kube-prometheus is having some problem on arm64.
Based on this issue prometheus-operator/kube-prometheus#545 it looks to be good now.
@ktsakalozos is there anyway we can try this on an arm arch?

@m-jayson
Copy link
Author

m-jayson commented Sep 17, 2020

@balchua thanks,

meanwhile, i tried accessing influxdb
https://192.168.1.150:16443/api/v1/namespaces/kube-system/services/monitoring-influxdb:http/proxy/

from
image

however it is giving me.
Error trying to reach service: 'dial tcp 10.1.97.129:8083: connect: connection refused'

but i can access grafana

@balchua
Copy link
Collaborator

balchua commented Sep 17, 2020

Fyi the influxdb and heapster were removed from MicroK8s 1.19 channel. The kube dashboard comes with the scraper and dashboard alone.

Coming back to your issue, what about port forwarding?

@m-jayson
Copy link
Author

@balchua scraper? sounds like prometheus to me? does that mean it has built in prometheus? how about grafana? will still be there?
if that is the case i'll update my version to 1.19 channel. but i need to verify it first

@balchua
Copy link
Collaborator

balchua commented Sep 17, 2020

Sorry i may have confused you. The dashboard scraper is not prometheus. It is part of the dashboard pods to get some metrics. It doesn't come with prometheus nor grafana.

The monitoring stack will still be the prometheus addon. Which is based on kube-prometheus.
Its unfortunate that i dont have an arm64 box to test it.

@m-jayson
Copy link
Author

i see. but let me verify my understanding.. so will there be prometheus on 1.19? how about grafana will still be there?. Sorry I'm still exploring microk8s and planning to stick with it

@m-jayson
Copy link
Author

@balchua if you could tell me what to do i could help you test it in arm64

@balchua
Copy link
Collaborator

balchua commented Sep 17, 2020

@mgonzaga1990 there is a file named /snap/microk8s/current/microk8s-resources/wrappers/addon-list.yaml where we specify whether the addon is available for which architecture.
For example:

    - name: "prometheus"
      description: "Prometheus operator for monitoring and logging"
      version: "v0.6.0"
      check_status: "pod/prometheus-k8s-0"
      supported_architectures:
      - amd64

But this file is read only. I don't know how to mount it to a writable filesystem.

@m-jayson
Copy link
Author

@balchua

I think that is different in my end.

image

cant see microk8s-resources

@m-jayson
Copy link
Author

@balchua i think as of now i may have to manually deploy prometheus in my cluster.. what image would you recommend for arm64?

@balchua
Copy link
Collaborator

balchua commented Sep 17, 2020

@mgonzaga1990 this is how we package the kube-prometheus https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/enable.prometheus.sh
Basically we just reduce the replicas.

@m-jayson
Copy link
Author

@balchua thanks for this.. i'm planning to upgrade now.. btw. what is the difference between classic and stable?

what would you recommend for raspberry pi?

@balchua
Copy link
Collaborator

balchua commented Sep 20, 2020

@mgonzaga1990 you can find here more information about the channels.

@m-jayson
Copy link
Author

@balchua i think i have to stick with 1.18 as of now.. i tried using the 1.19 but at some point my raspberry pi freezes when running pipeline (drone and gitea) which i don't experience with 1.18

@balchua
Copy link
Collaborator

balchua commented Sep 21, 2020

If it is not too much, will it be possible to get the inspect tarball? Will really help to troubleshoot it.

@m-jayson
Copy link
Author

@balchua sure sure.. i'll geenrate it and let you know once available

@balchua
Copy link
Collaborator

balchua commented Sep 22, 2020

@mgonzaga1990 perhaps opening a new issue with the 1.19 freezing up your rpis will be a good way to track that. Wdyt?

@m-jayson
Copy link
Author

m-jayson commented Sep 22, 2020 via email

@nderraugh
Copy link

nderraugh commented Oct 11, 2020

I have the same problem on the 1.19 channel. I'm on a pi 4b and ubuntu 20.04.1.

I think the arm64 issue may be a red herring. I am new to microk8s and snap, but looking through /snap/microk8s/1669/microk8s-enable.wrapper it looks like there is neither a "${SNAP}/actions/enable.$action.sh" nor a "${SNAP}/actions/$action.yaml" for prometheus. Hence the Nothing to do for prometheus message.

I ran snap run --shell microk8s.status to set my env vars and then ran /snap/microk8s/1669/microk8s-resources/actions/enable.prometheus.sh and made considerable progress.

Here's the output:

Enabling DNS
Applying manifest
serviceaccount/coredns unchanged
configmap/coredns unchanged
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
DNS is enabled
Fetching kube-prometheus version v0.6.0.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   143  100   143    0     0    242      0 --:--:-- --:--:-- --:--:--   242
100  267k    0  267k    0     0   156k      0 --:--:--  0:00:01 --:--:-- 1342k
kube-prometheus-0.6.0/
kube-prometheus-0.6.0/.github/
kube-prometheus-0.6.0/.github/ISSUE_TEMPLATE/
kube-prometheus-0.6.0/.github/ISSUE_TEMPLATE/bug.md
kube-prometheus-0.6.0/.github/ISSUE_TEMPLATE/feature.md
kube-prometheus-0.6.0/.github/ISSUE_TEMPLATE/support.md
kube-prometheus-0.6.0/.gitignore
kube-prometheus-0.6.0/.travis.yml
kube-prometheus-0.6.0/DCO
kube-prometheus-0.6.0/LICENSE
kube-prometheus-0.6.0/Makefile
kube-prometheus-0.6.0/NOTICE
kube-prometheus-0.6.0/OWNERS
kube-prometheus-0.6.0/README.md
kube-prometheus-0.6.0/build.sh
kube-prometheus-0.6.0/code-of-conduct.md
kube-prometheus-0.6.0/docs/
kube-prometheus-0.6.0/docs/EKS-cni-support.md
kube-prometheus-0.6.0/docs/GKE-cadvisor-support.md
kube-prometheus-0.6.0/docs/developing-prometheus-rules-and-grafana-dashboards.md
kube-prometheus-0.6.0/docs/exposing-prometheus-alertmanager-grafana-ingress.md
kube-prometheus-0.6.0/docs/kube-prometheus-on-kubeadm.md
kube-prometheus-0.6.0/docs/monitoring-external-etcd.md
kube-prometheus-0.6.0/docs/monitoring-other-namespaces.md
kube-prometheus-0.6.0/docs/weave-net-support.md
kube-prometheus-0.6.0/example.jsonnet
kube-prometheus-0.6.0/examples/
kube-prometheus-0.6.0/examples/additional-namespaces-servicemonitor.jsonnet
kube-prometheus-0.6.0/examples/additional-namespaces.jsonnet
kube-prometheus-0.6.0/examples/alertmanager-config-external.jsonnet
kube-prometheus-0.6.0/examples/alertmanager-config.jsonnet
kube-prometheus-0.6.0/examples/alertmanager-config.yaml
kube-prometheus-0.6.0/examples/auth
kube-prometheus-0.6.0/examples/basic-auth/
kube-prometheus-0.6.0/examples/basic-auth/secrets.yaml
kube-prometheus-0.6.0/examples/basic-auth/service-monitor.yaml
kube-prometheus-0.6.0/examples/eks-cni-example.jsonnet
kube-prometheus-0.6.0/examples/etcd-client-ca.crt
kube-prometheus-0.6.0/examples/etcd-client.crt
kube-prometheus-0.6.0/examples/etcd-client.key
kube-prometheus-0.6.0/examples/etcd-skip-verify.jsonnet
kube-prometheus-0.6.0/examples/etcd.jsonnet
kube-prometheus-0.6.0/examples/example-app/
kube-prometheus-0.6.0/examples/example-app/example-app.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-alertmanager-discovery-role-binding.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-alertmanager-discovery-role.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-role-binding.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-role.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-service-account.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-svc.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend.yaml
kube-prometheus-0.6.0/examples/example-app/servicemonitor-frontend.yaml
kube-prometheus-0.6.0/examples/example-grafana-dashboard.json
kube-prometheus-0.6.0/examples/existingrule.json
kube-prometheus-0.6.0/examples/existingrule.yaml
kube-prometheus-0.6.0/examples/grafana-additional-jsonnet-dashboard-example.jsonnet
kube-prometheus-0.6.0/examples/grafana-additional-rendered-dashboard-example-2.jsonnet
kube-prometheus-0.6.0/examples/grafana-additional-rendered-dashboard-example.jsonnet
kube-prometheus-0.6.0/examples/ingress.jsonnet
kube-prometheus-0.6.0/examples/internal-registry.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-build-snippet/
kube-prometheus-0.6.0/examples/jsonnet-build-snippet/build-snippet.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/
kube-prometheus-0.6.0/examples/jsonnet-snippets/bootkube.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/kops-coredns.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/kops.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/kube-aws.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/kubeadm.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/kubespray.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/node-ports.jsonnet
kube-prometheus-0.6.0/examples/ksonnet-example.jsonnet
kube-prometheus-0.6.0/examples/kustomize.jsonnet
kube-prometheus-0.6.0/examples/minikube.jsonnet
kube-prometheus-0.6.0/examples/prometheus-additional-alert-rule-example.jsonnet
kube-prometheus-0.6.0/examples/prometheus-additional-recording-rule-example.jsonnet
kube-prometheus-0.6.0/examples/prometheus-additional-rendered-rule-example.jsonnet
kube-prometheus-0.6.0/examples/prometheus-name-override.jsonnet
kube-prometheus-0.6.0/examples/prometheus-pvc.jsonnet
kube-prometheus-0.6.0/examples/tolerations.libsonnet
kube-prometheus-0.6.0/examples/weave-net-example.jsonnet
kube-prometheus-0.6.0/experimental/
kube-prometheus-0.6.0/experimental/metrics-server/
kube-prometheus-0.6.0/experimental/metrics-server/auth-delegator.yaml
kube-prometheus-0.6.0/experimental/metrics-server/auth-reader.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-apiservice.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-server-cluster-role-binding.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-server-cluster-role.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-server-deployment.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-server-service-account.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-server-service.yaml
kube-prometheus-0.6.0/go.mod
kube-prometheus-0.6.0/go.sum
kube-prometheus-0.6.0/hack/
kube-prometheus-0.6.0/hack/example-service-monitoring/
kube-prometheus-0.6.0/hack/example-service-monitoring/deploy
kube-prometheus-0.6.0/hack/example-service-monitoring/teardown
kube-prometheus-0.6.0/hack/jsonnet-docker-image
kube-prometheus-0.6.0/jsonnet/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/.gitignore
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alertmanager/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alertmanager/alertmanager.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/alertmanager.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/alerts.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/general.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/node.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/prometheus-operator.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/tests.yaml
kube-prometheus-0.6.0/jsonnet/kube-prometheus/dropping-deprecated-metrics-relabelings.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/grafana-weave-net-cluster.json
kube-prometheus-0.6.0/jsonnet/kube-prometheus/grafana-weave-net.json
kube-prometheus-0.6.0/jsonnet/kube-prometheus/jsonnetfile.json
kube-prometheus-0.6.0/jsonnet/kube-prometheus/ksm-autoscaler/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/ksm-autoscaler/ksm-autoscaler.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-all-namespaces.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-anti-affinity.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-bootkube.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-config-mixins.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-custom-metrics.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-eks.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-insecure-kubelet.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-kops-coredns.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-kops.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-ksonnet.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-kube-aws.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-kubeadm.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-kubespray.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-managed-cluster.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-node-ports.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-static-etcd.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-strip-limits.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-weave-net.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-rbac-proxy/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-rbac-proxy/container.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-state-metrics/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-state-metrics/kube-state-metrics.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/lib/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/lib/image.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/lib/lib.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/node-exporter/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/node-exporter/node-exporter.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/prometheus-adapter/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/prometheus-adapter/prometheus-adapter.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/prometheus/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/prometheus/prometheus.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/rules/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/rules/general.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/rules/node-rules.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/rules/rules.libsonnet
kube-prometheus-0.6.0/jsonnetfile.json
kube-prometheus-0.6.0/jsonnetfile.lock.json
kube-prometheus-0.6.0/kustomization.yaml
kube-prometheus-0.6.0/manifests/
kube-prometheus-0.6.0/manifests/alertmanager-alertmanager.yaml
kube-prometheus-0.6.0/manifests/alertmanager-secret.yaml
kube-prometheus-0.6.0/manifests/alertmanager-service.yaml
kube-prometheus-0.6.0/manifests/alertmanager-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/alertmanager-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/grafana-dashboardDatasources.yaml
kube-prometheus-0.6.0/manifests/grafana-dashboardDefinitions.yaml
kube-prometheus-0.6.0/manifests/grafana-dashboardSources.yaml
kube-prometheus-0.6.0/manifests/grafana-deployment.yaml
kube-prometheus-0.6.0/manifests/grafana-service.yaml
kube-prometheus-0.6.0/manifests/grafana-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/grafana-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-clusterRole.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-clusterRoleBinding.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-deployment.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-service.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/node-exporter-clusterRole.yaml
kube-prometheus-0.6.0/manifests/node-exporter-clusterRoleBinding.yaml
kube-prometheus-0.6.0/manifests/node-exporter-daemonset.yaml
kube-prometheus-0.6.0/manifests/node-exporter-service.yaml
kube-prometheus-0.6.0/manifests/node-exporter-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/node-exporter-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-apiService.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-clusterRole.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-clusterRoleBinding.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-clusterRoleBindingDelegator.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-clusterRoleServerResources.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-configMap.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-deployment.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-roleBindingAuthReader.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-service.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/prometheus-clusterRole.yaml
kube-prometheus-0.6.0/manifests/prometheus-clusterRoleBinding.yaml
kube-prometheus-0.6.0/manifests/prometheus-operator-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/prometheus-prometheus.yaml
kube-prometheus-0.6.0/manifests/prometheus-roleBindingConfig.yaml
kube-prometheus-0.6.0/manifests/prometheus-roleBindingSpecificNamespaces.yaml
kube-prometheus-0.6.0/manifests/prometheus-roleConfig.yaml
kube-prometheus-0.6.0/manifests/prometheus-roleSpecificNamespaces.yaml
kube-prometheus-0.6.0/manifests/prometheus-rules.yaml
kube-prometheus-0.6.0/manifests/prometheus-service.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitorApiserver.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitorCoreDNS.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitorKubeControllerManager.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitorKubeScheduler.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitorKubelet.yaml
kube-prometheus-0.6.0/manifests/setup/
kube-prometheus-0.6.0/manifests/setup/0namespace-namespace.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0podmonitorCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0prometheusruleCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0servicemonitorCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0thanosrulerCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-clusterRole.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-clusterRoleBinding.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-deployment.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-service.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-serviceAccount.yaml
kube-prometheus-0.6.0/scripts/
kube-prometheus-0.6.0/scripts/minikube-start-kvm.sh
kube-prometheus-0.6.0/scripts/minikube-start.sh
kube-prometheus-0.6.0/scripts/monitoring-deploy.sh
kube-prometheus-0.6.0/scripts/tools.go
kube-prometheus-0.6.0/sync-to-internal-registry.jsonnet
kube-prometheus-0.6.0/test.sh
kube-prometheus-0.6.0/tests/
kube-prometheus-0.6.0/tests/e2e/
kube-prometheus-0.6.0/tests/e2e/main_test.go
kube-prometheus-0.6.0/tests/e2e/prometheus_client.go
kube-prometheus-0.6.0/tests/e2e/travis-e2e.sh
Enabling Prometheus
namespace/monitoring created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created
alertmanager.monitoring.coreos.com/main created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
servicemonitor.monitoring.coreos.com/alertmanager created
secret/grafana-datasources created
configmap/grafana-dashboard-apiserver created
configmap/grafana-dashboard-cluster-total created
configmap/grafana-dashboard-controller-manager created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-node created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-kubelet created
configmap/grafana-dashboard-namespace-by-pod created
configmap/grafana-dashboard-namespace-by-workload created
configmap/grafana-dashboard-node-cluster-rsrc-use created
configmap/grafana-dashboard-node-rsrc-use created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pod-total created
configmap/grafana-dashboard-prometheus-remote-write created
configmap/grafana-dashboard-prometheus created
configmap/grafana-dashboard-proxy created
configmap/grafana-dashboard-scheduler created
configmap/grafana-dashboard-statefulset created
configmap/grafana-dashboard-workload-total created
configmap/grafana-dashboards created
deployment.apps/grafana created
service/grafana created
serviceaccount/grafana created
servicemonitor.monitoring.coreos.com/grafana created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
service/node-exporter created
serviceaccount/node-exporter created
servicemonitor.monitoring.coreos.com/node-exporter created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
servicemonitor.monitoring.coreos.com/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus-operator created
prometheus.monitoring.coreos.com/k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created
The Prometheus operator is enabled (user/pass: admin/admin)

In the pods that were created there are still several problems. Specifically the logs of several crashing pods mention exec format error (related to the arch?). But I think if somone who knows snap and microk8s better than me can connect some dots here we can resolve the original "Nothing to do for Prometheus" issue and look at solving these pod failures if they still exist.

@nderraugh
Copy link

nderraugh commented Oct 11, 2020

Following up from the link to prometheus-operator/kube-prometheus#545 I removed microk8s 1.19 and intalled 1.18 (I hadn't noticed that kube-prometheus support for 1.19 was still at HEAD and not 0.6). I then followed the same procedure as I outlined in my previous post and it failed with error: the path "/snap/microk8s/1705/actions/prometheus/setup" does not exist.

I ran through the commands in enable.prometheus.sh by hand and it fails at $KUBECTL apply -f "${SNAP}/actions/prometheus/setup". I changed it to $KUBECTL apply -f "${SNAP}/microk8s-resources/actions/prometheus/setup" and it ran successfully.

namespace/monitoring created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created

The next step failed with the same path problem. But I changed it to ($KUBECTL apply -f "${SNAP}/microk8s-resources/actions/prometheus/") and it ran successfully.

alertmanager.monitoring.coreos.com/main created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
servicemonitor.monitoring.coreos.com/alertmanager created
secret/grafana-datasources created
configmap/grafana-dashboard-apiserver created
configmap/grafana-dashboard-cluster-total created
configmap/grafana-dashboard-controller-manager created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-node created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-kubelet created
configmap/grafana-dashboard-namespace-by-pod created
configmap/grafana-dashboard-namespace-by-workload created
configmap/grafana-dashboard-node-cluster-rsrc-use created
configmap/grafana-dashboard-node-rsrc-use created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pod-total created
configmap/grafana-dashboard-pods created
configmap/grafana-dashboard-prometheus-remote-write created
configmap/grafana-dashboard-prometheus created
configmap/grafana-dashboard-proxy created
configmap/grafana-dashboard-scheduler created
configmap/grafana-dashboard-statefulset created
configmap/grafana-dashboard-workload-total created
configmap/grafana-dashboards created
deployment.apps/grafana created
service/grafana created
serviceaccount/grafana created
servicemonitor.monitoring.coreos.com/grafana created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
role.rbac.authorization.k8s.io/kube-state-metrics created
rolebinding.rbac.authorization.k8s.io/kube-state-metrics created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
service/node-exporter created
serviceaccount/node-exporter created
servicemonitor.monitoring.coreos.com/node-exporter created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus-operator created
prometheus.monitoring.coreos.com/k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created

The pods that start up still have the same exec format eror in their logs though, except for grafana which has this error 2020-10-11T16:31:06.142685100Z /run.sh: line 80: /usr/share/grafana/bin/grafana-server: No such file or directory.

FWIW, cloning the kube-prometheus repo and running through the Quickstart resulted in prometheus-operator coming up successfully, followed by node-exporter, grafana, prometheus-adapter, alert-manager-main-{0,1,2}, prometheus-k8s-{0,1}. However the kube-state-metrics container still fails with exec format error in the logs.

@nderraugh
Copy link

As a workaround on 1.19 you can do as suggested in kubernetes/kube-state-metrics#1190 (comment) and modify https://github.com/prometheus-operator/kube-prometheus/blob/980e95de011319b88a3b9c0787a81dcdf338a898/manifests/kube-state-metrics-deployment.yaml#L26 from

image: quay.io/coreos/kube-state-metrics:v1.9.7

to

image: gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-arm64:v1.9.7

and everything comes up as expected.

@seangustafson
Copy link

I can confirm that the prometheus add-on installed successfully in microk8s 1.20 on my raspberry pi cluster running Ubuntu 20.04.1 (like the others in this thread I had problems with 1.19). Given that, I think this issue can now be closed.

@balchua
Copy link
Collaborator

balchua commented Dec 12, 2020

Thanks @seangustafson, this addon also works on mixed clusters. I.e a mix of arm64 and amd64 nodes.

@balchua balchua closed this as completed Dec 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants