Improvement/use node exporter pkg #113

Zempashi · 2018-07-04T15:57:48Z

Fixes: #31
Fixes: #56

Zempashi · 2018-07-04T15:58:57Z

roles/node_exporter/tasks/main.yml

+    repo_gpgcheck: True
+    state: present
+
+# https://github.com/ansible/ansible/issues/20711


@ballot-scality I'm not sure how to detect that the key has been accepted or not.

Zempashi · 2018-07-04T16:01:44Z

tests/single-node/test.sh

@@ -103,3 +105,17 @@ EOF
        assert_fails 'make_shell kubectl get pv -o jsonpath={.items[*].status.phase} | grep Released > /dev/null' \
                "PVs in Released state found"
 }
+
+test_prometheus_node_exporter_metrics() {


This test rely on being on the server itself (yum and accessing prometheus cluster IP directly) but test are better than no test. As already discussed, we might want to move to a more feature-full framework (list of features to be defined)

NicolasT · 2018-07-04T16:02:12Z

metal-k8s.yml

    - role: kube_heapster
      tags: ['heapster']
    - role: kube_elasticsearch
      tags: ['elasticsearch']
    - role: kube_metrics_server
      tags: ['metrics-server']

+- hosts: kube-node


Shouldn't this go on kube-node, kube-master (so actually k8s-cluster) and etcd?

NicolasT · 2018-07-04T16:05:03Z

roles/kube_prometheus_node_exporter/tasks/main.yml

+- name: 'make assertions'
+  assert:
+    that:
+     - prometheus_addon_dir is defined


Assertion comes after first usage (above), which is odd?

NicolasT · 2018-07-04T16:08:02Z

playbooks/services.yml

    - role: kube_heapster
      tags: ['heapster']
    - role: kube_elasticsearch
      tags: ['elasticsearch']
      when: metalk8s_elasticsearch_enabled|bool
    - role: kube_metrics_server
      tags: ['metrics-server']
+
+- hosts: kube-node


Shouldn't this be k8s-cluster:etcd?

NicolasT · 2018-07-04T16:08:32Z

roles/node_exporter/tasks/main.yml

+    baseurl: https://packagecloud.io/prometheus-rpm/release/el/$releasever/$basearch
+    description: prometheus-rpm_release
+    enabled: True
+    gpgcheck: False


but repo_gpgcheck: True (which is an oddity specific to packagecloud.io)

NicolasT · 2018-07-04T16:08:51Z

roles/node_exporter/tasks/main.yml

+    name: prometheus-rpm_release
+    baseurl: https://packagecloud.io/prometheus-rpm/release/el/$releasever/$basearch
+    description: prometheus-rpm_release
+    enabled: True


Maybe keep disabled by default and explicitly enable when installing node_exporter?

For update sake ? Talking to @ballot-scality, there is some plugin to install yum-version-lock. Might be better

Update would all of a sudden install 0.16 :) I'm not too fond of version-locking myself. I believe Kubespray even uses a separate yum.conf for Docker installs to keep things separated. You can enable repos on yum invocation (so likely also through the yum module)

repo docker are enabled (and yes install docker-ce will install 18.03, which is bad).
We must avoid doing the same mistake of kubespray. Version locking could fix docker problem (because disabling docker repo after kubespray will not prevent kubespray itself to reenable it).
It will also allow some upgrade and prevent what we have on the ring multiple times: "oh the package is now on epel so the system decide to take the version of epel instead of our repo."

Fair enough 😃

NicolasT

Didn't test, but LGTM on-sight...

Looking into making the packagecloud.io repo not enabled by default could be nice, but could be a follow-up PR as far as I'm concerned.

Also, a bit surprised you need the makecache, I thought yum does that automatically when it finds new repositories.

NicolasT · 2018-07-04T16:33:54Z

CI seems unhappy 🤖

NicolasT · 2018-07-04T16:58:07Z

tests/single-node/test.sh

+          jq '.data.activeTargets |[.[]|select(.labels.job == $job)]|length' \
+          --arg job "node-exporter")
+        echo "Found ${NB_TARGET} targets"
+        assert [ "${NB_TARGET}" -gt "0" ]


For future:

assert_not_equals 0 ${NB_TARGET}

NicolasT · 2018-07-04T17:40:18Z

Fixed CI by applying #113 (comment)

NicolasT

I assume this has been tested, and existing Grafana dashboards still work 😃

See: #113

Zempashi · 2018-07-05T07:49:18Z

I've checked that when reverting from 0.16 to 0.15 on one node, some dashboards where back. I'll double check

See: c73b5f3 See: ansible/ansible#20711 See: #113 See: #113 See: #31 See: #31

Fixes: https://github.com/scality/metal-k8s/projects/1#card-11097887 See: #31 See: #31 See: #113 See: #113 See: prometheus-operator/prometheus-operator#1560

Zempashi commented Jul 4, 2018

View reviewed changes

NicolasT added kind:enhancement New feature or request topic:operations Operations-related issues labels Jul 4, 2018

NicolasT added this to the MetalK8s 0.1.0 milestone Jul 4, 2018

NicolasT requested review from a team and ballot-scality July 4, 2018 16:00

Zempashi commented Jul 4, 2018

View reviewed changes

Julien Girardin and others added 2 commits July 4, 2018 18:02

ZENKO-257: Remove node_exporter deployment via kube-prometheus

5b7b953

ZENKO-257: Deploy node_exporter for prometheus as a system package

c73b5f3

Zempashi force-pushed the improvement/use_node_exporter_pkg branch from 43af70d to 85d7952 Compare July 4, 2018 16:06

NicolasT reviewed Jul 4, 2018

View reviewed changes

ZENKO-257: kube_prometheus_node_exporter for kubernetes manifests setup

a70d9aa

Zempashi force-pushed the improvement/use_node_exporter_pkg branch from 85d7952 to 0c1446b Compare July 4, 2018 16:12

ZENKO-257: Wire node_exporter roles into playbook

2579bed

Zempashi force-pushed the improvement/use_node_exporter_pkg branch from 0c1446b to 8ac8b42 Compare July 4, 2018 16:14

NicolasT previously approved these changes Jul 4, 2018

View reviewed changes

Zempashi dismissed NicolasT’s stale review via 7d4c9ab July 4, 2018 16:49

Zempashi force-pushed the improvement/use_node_exporter_pkg branch from 8ac8b42 to 7d4c9ab Compare July 4, 2018 16:49

NicolasT reviewed Jul 4, 2018

View reviewed changes

ZENKO-257: Unit test to list target of node-exporter

1c15210

NicolasT force-pushed the improvement/use_node_exporter_pkg branch from 7d4c9ab to 1c15210 Compare July 4, 2018 17:20

NicolasT previously approved these changes Jul 4, 2018

View reviewed changes

Zempashi dismissed NicolasT’s stale review via 7ec5ef7 July 4, 2018 19:30

Zempashi force-pushed the improvement/use_node_exporter_pkg branch 2 times, most recently from d478cdd to 2d35169 Compare July 4, 2018 20:13

NicolasT previously approved these changes Jul 4, 2018

View reviewed changes

ZENKO-257: Lock version of node_exporter

f7d606f

Zempashi dismissed NicolasT’s stale review via f7d606f July 5, 2018 06:40

Zempashi force-pushed the improvement/use_node_exporter_pkg branch from 2d35169 to f7d606f Compare July 5, 2018 06:40

NicolasT approved these changes Jul 5, 2018

View reviewed changes

NicolasT added a commit that referenced this pull request Jul 5, 2018

doc: Add host-based node_exporter to ChangeLog

7277596

See: #113

Zempashi merged commit ce42294 into development/0.1 Jul 5, 2018

NicolasT deleted the improvement/use_node_exporter_pkg branch July 5, 2018 08:04

NicolasT added a commit that referenced this pull request Jul 9, 2018

node_exporter: Enable repository GPG keys

23f17d9

See: c73b5f3 See: ansible/ansible#20711 See: #113 See: #113 See: #31 See: #31

NicolasT mentioned this pull request Jul 9, 2018

kube_prometheus: Use upstream 'external node_exporter' feature #133

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvement/use node exporter pkg #113

Improvement/use node exporter pkg #113

Zempashi commented Jul 4, 2018 •

edited by NicolasT

Loading

Zempashi Jul 4, 2018

Zempashi Jul 4, 2018

NicolasT Jul 4, 2018

NicolasT Jul 4, 2018

NicolasT Jul 4, 2018

NicolasT Jul 4, 2018

Zempashi Jul 4, 2018

NicolasT Jul 4, 2018

Zempashi Jul 4, 2018

NicolasT Jul 4, 2018

Zempashi Jul 4, 2018

NicolasT Jul 4, 2018

NicolasT left a comment

NicolasT commented Jul 4, 2018

NicolasT Jul 4, 2018

NicolasT commented Jul 4, 2018

NicolasT left a comment

Zempashi commented Jul 5, 2018

Improvement/use node exporter pkg #113

Improvement/use node exporter pkg #113

Conversation

Zempashi commented Jul 4, 2018 • edited by NicolasT Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NicolasT left a comment

Choose a reason for hiding this comment

NicolasT commented Jul 4, 2018

Choose a reason for hiding this comment

NicolasT commented Jul 4, 2018

NicolasT left a comment

Choose a reason for hiding this comment

Zempashi commented Jul 5, 2018

Zempashi commented Jul 4, 2018 •

edited by NicolasT

Loading