[etcd] Use prometheus metricset for etcd v3 metrics #8438

gpop63 · 2023-11-08T21:40:50Z

Overview

Changed the data stream metricset to prometheus to ensure we can update and manage the data stream independently of the etcd beats module releases.

Other changes:

Added ECS fields
Added metric types and dimensions
Added a pipeline to rename fields
Added etcd labels and a labels fingerprint for TSDB purposes

TSDB test

Testing data stream metrics-etcd.metrics-default.
Index being used for the documents is .ds-metrics-etcd.metrics-default-2023.11.08-000001.
Index being used for the settings and mappings is .ds-metrics-etcd.metrics-default-2023.11.08-000001.

The time series fields for the TSDB index are: 
        - dimension (10 fields):
                - agent.id
                - cloud.account.id
                - cloud.availability_zone
                - cloud.instance.id
                - cloud.provider
                - cloud.region
                - container.id
                - etcd.labels.fingerprint
                - host.name
                - service.address
        - counter (7 fields):
                - etcd.memory.go_memstats_alloc.total.bytes
                - etcd.network.client_grpc_received.bytes
                - etcd.network.client_grpc_sent.bytes
                - etcd.server.grpc_handled.count
                - etcd.server.grpc_started.count
                - etcd.server.leader_changes.count
                - etcd.server.proposals_failed.count
        - gauge (5 fields):
                - etcd.disk.mvcc_db_total_size.bytes
                - etcd.memory.go_memstats_alloc.bytes
                - etcd.server.has_leader.count
                - etcd.server.proposals_committed.count
                - etcd.server.proposals_pending.count
        - routing_path (10 fields):
                - agent.id
                - cloud.account.id
                - cloud.availability_zone
                - cloud.instance.id
                - cloud.provider
                - cloud.region
                - container.id
                - etcd.labels.fingerprint
                - host.name
                - service.address

Index tsdb-index-enabled successfully created.

Copying documents from .ds-metrics-etcd.metrics-default-2023.11.08-000001 to tsdb-index-enabled...
All 23350 documents taken from index .ds-metrics-etcd.metrics-default-2023.11.08-000001 were successfully placed to index tsdb-index-enabled.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

elastic-package stack up -d -v --version 8.12.0-SNAPSHOT

docker-compose.yml

version: '3'
services:
  etcd: 
    image: bitnami/etcd:3.5.1
    container_name: production_etcd
    environment:
      - ALLOW_NONE_AUTHENTICATION=yes
    ports:
      - 4001:4001
      - 2380:2380
      - 2379:2379
  elastic-agent:
    image: docker.elastic.co/beats/elastic-agent:8.12.0-SNAPSHOT
    container_name: elastic-agent
    restart: always
    user: root
    environment:
      - FLEET_ENROLLMENT_TOKEN=TOKEN
      - FLEET_ENROLL=1
      - FLEET_INSECURE=1
      - FLEET_URL=https://fleet-server:8220
networks:
  default:
    external: true
    name: elastic-package-stack_default

Related issues

Screenshots

change input type improve titles and descriptions

elasticmachine · 2023-11-08T21:44:03Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-11-30T12:45:58.443+0000
Duration: 17 min 56 sec

Test stats 🧪

Test	Results
Failed	0
Passed	12
Skipped	0
Total	12

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-11-08T21:59:03Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚 5.556
Classes	100.0% (`0/0`)	💚 5.556
Methods	60.0% (`9/15`)	👎 -29.701
Lines	100.0% (`0/0`)	💚 13.917
Conditionals	100.0% (`0/0`)	💚

agithomas · 2023-11-09T06:06:46Z

Can you update the section Compatability highlighting the compatibility metrics ? It may be best to highlight that if the etcd version is 2, dataset named metrics is not supported.

@lalit-satapathy / @SubhrataK ,

We have three datasets based on V2 API - leader, self and store. Dataset named metrics datastream is based on V3 API.

If etcd version is above 3, by default it supports V3 API. But, It support V2 APIs (by setting ENV variables). So V2 & V3 APIs are supported in etcd version 3 and above. I have tested it with 3.5 version of etcd.

The issue here is, V2 APIs provide more metrics when compared to V3 APIs . So, by not supporting V2 API based metrics, customer may loose important insights. @gpop63 please correct me if am wrong.

Considering the above points, even when etcd version 2.x is not actively maintained, having V2 API continue to be supported in etcd version 3 and above, should we continue to support datasets leader, self and store in etcd GA version? I think, we should. Like to hear your inputs as well.

agithomas · 2023-11-09T06:10:02Z

Based on the above comment, I think, we should do the TSDB enablement of all datasets .

packages/etcd/data_stream/metrics/fields/fields.yml

agithomas

Added review comments

gpop63 · 2023-11-09T11:41:20Z

@agithomas The use of etcd v2 is highly discouraged, I think it's not worth spending effort on it.

To my knowledge, v2 APIs can be enabled in etcd v3.5, but they are completely removed from v3.6 onwards.

Are you suggesting that this data stream should also capture v2 specific metrics from etcd v3 instances where the v2 APIs are enabled?

agithomas · 2023-11-09T11:44:40Z

To my knowledge, v2 APIs can be enabled in etcd v3.5, but they are completely removed from v3.6 onwards.

I see that 3.6 is in draft state and 3.5 is the latest stable release. Can you share the link to source of info that says V2 APIs will be deprecated from 3.6 ? This will help the decision making process faster

agithomas · 2023-11-09T11:46:22Z

Are you suggesting that this data stream should also capture v2 specific metrics from etcd v3 instances where the v2 APIs are enabled?

I am evaluating the benefit of retaining the existing datasets leader, self and store instead of deprecating or removing them as part of GA release.

gpop63 · 2023-11-09T11:46:44Z

Can you share the link to source of info that says V2 APIs will be deprecated from 3.6 ? This will help the decision making process faster

etcd-io/etcd#12913

agithomas · 2023-11-19T04:39:06Z

packages/etcd/manifest.yml

-        description: Collecting etcd metrics
+        title: Collect metrics from etcd v2 instances
+        description: Collecting metrics etcd v2 metrics
+      - type: prometheus/metrics


Is this really necessary as both v2 and v3 endpoint prefix are the same ?

Should we have 1 global hosts var instead?

Yes, that will serve the purpose as the prefix of both endpoints remain the same.

packages/etcd/data_stream/metrics/fields/package-fields.yml

agithomas · 2023-11-19T06:46:09Z

packages/etcd/data_stream/metrics/fields/fields.yml

-              description: |
-                Memory allocated bytes as of MemStats Go
+        - name: go_memstats_alloc.bytes
+          type: long


Don't you want to put unit: byte and similar for field mappings?

gpop63 · 2023-11-20T13:17:30Z

@agithomas from the new metrics added, which ones do you think should we add in the dashboard?

gpop63 · 2023-11-20T13:17:58Z

/test

agithomas · 2023-11-27T02:49:48Z

@ritalwar , can you please take a look at the PR and do a review? Kibana visualisation changes are suggested by me which is pending. You may look at other changes except this. Thanks in advance.

packages/etcd/changelog.yml

packages/etcd/data_stream/metrics/sample_event.json

gpop63 · 2023-11-27T21:47:10Z

Added data stream filter
Improved lens titles, metric values format e.g. bytes.
Added Storage Metrics and Proposals Applied lenses

Before:

After:

Co-authored-by: Richa Talwar <102972658+ritalwar@users.noreply.github.com>

agithomas · 2023-11-29T05:52:01Z

@gpop63 , can you have the panels organised in a way that

Panels 1,2,3 are in one row. 10 and 11 in another, 20 and 21 in another ? Please refer to the below mentioned screenshot.

Also, please update the package screenshot image as well.

gpop63 · 2023-11-29T14:46:02Z

Updated:

agithomas

LGTM!

agithomas · 2023-11-29T14:49:46Z

@ritalwar , can you please take a final look at the PR ?

packages/etcd/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml

packages/etcd/data_stream/metrics/fields/fields.yml

packages/etcd/data_stream/metrics/fields/package-fields.yml

ritalwar

LGTM!

elasticmachine · 2023-11-30T19:58:58Z

Package etcd - 0.7.0 containing this change is available at https://epr.elastic.co/search?package=etcd

gpop63 added 5 commits November 8, 2023 23:36

add docker healthcheck

28be9c3

remove vars from test

1f1cb0b

use prometheus metricset

8f54107

update docs

bdffc5f

bump package version

42cab95

change input type improve titles and descriptions

gpop63 force-pushed the update_etcd_v3_metrics branch from adab794 to 42cab95 Compare November 8, 2023 21:41

gpop63 requested a review from agithomas November 8, 2023 22:16

fix sample eventt

5e6ed13

gpop63 force-pushed the update_etcd_v3_metrics branch from 96c28d3 to 5e6ed13 Compare November 8, 2023 22:48

agithomas reviewed Nov 9, 2023

View reviewed changes

packages/etcd/data_stream/metrics/fields/fields.yml Show resolved Hide resolved

agithomas reviewed Nov 9, 2023

View reviewed changes

This was referenced Nov 9, 2023

[meta] [etcd] GA Migration #8441

Closed

[etcd] Package enhancement to use prometheus integration package and update ETCD compatibility information #8445

Closed

lalit-satapathy mentioned this pull request Nov 15, 2023

[etcd] Add metric type mapping for metrics dataset #8351

Closed

4 tasks

gpop63 added 5 commits November 19, 2023 01:20

disable v2 data streams by default

46a6ec4

add additional v3 metrics

1e20424

fix etcd policy template

135a199

update docs

43b3ffd

Merge remote-tracking branch 'upstream/main' into update_etcd_v3_metrics

80f8970

agithomas reviewed Nov 19, 2023

View reviewed changes

agithomas requested a review from ritalwar November 27, 2023 02:48

ritalwar reviewed Nov 27, 2023

View reviewed changes

packages/etcd/changelog.yml Outdated Show resolved Hide resolved

packages/etcd/data_stream/metrics/sample_event.json Outdated Show resolved Hide resolved

gpop63 added 2 commits November 27, 2023 23:34

set ecs version in pipeline

b932989

update dashboard

99bec2d

gpop63 and others added 4 commits November 27, 2023 23:47

Update packages/etcd/changelog.yml

f04ce51

Co-authored-by: Richa Talwar <102972658+ritalwar@users.noreply.github.com>

bump kibana version

044cdb4

change migration version

76423b6

export with 8.7

39bbfb9

gpop63 requested review from agithomas and ritalwar November 28, 2023 10:31

gpop63 added 2 commits November 29, 2023 14:04

reoder lenses

6eb4db3

replace dashboard image

03f1a7c

agithomas approved these changes Nov 29, 2023

View reviewed changes

agithomas requested a review from muthu-mps November 29, 2023 14:50

ritalwar reviewed Nov 30, 2023

View reviewed changes

packages/etcd/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

ritalwar reviewed Nov 30, 2023

View reviewed changes

packages/etcd/data_stream/metrics/fields/fields.yml Outdated Show resolved Hide resolved

packages/etcd/data_stream/metrics/fields/fields.yml Outdated Show resolved Hide resolved

packages/etcd/data_stream/metrics/fields/package-fields.yml Show resolved Hide resolved

gpop63 added 3 commits November 30, 2023 12:00

fix ecs version in pipeline

d8a27a1

update fields description

f35b8b7

update sample event

49ed975

ritalwar approved these changes Nov 30, 2023

View reviewed changes

update docs

679064b

gpop63 merged commit 131e7e7 into elastic:main Nov 30, 2023
4 checks passed

This was referenced Dec 1, 2023

[etcd] TSDB Enablement #8327

Closed

[etcd] Enable TSDB for metrics datastream #8649

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[etcd] Use prometheus metricset for etcd v3 metrics #8438

[etcd] Use prometheus metricset for etcd v3 metrics #8438

gpop63 commented Nov 8, 2023 •

edited

elasticmachine commented Nov 8, 2023 •

edited

Build stats

Test stats 🧪

elasticmachine commented Nov 8, 2023 •

edited

agithomas commented Nov 9, 2023

agithomas commented Nov 9, 2023

agithomas left a comment

gpop63 commented Nov 9, 2023

agithomas commented Nov 9, 2023

agithomas commented Nov 9, 2023

gpop63 commented Nov 9, 2023

agithomas Nov 19, 2023

gpop63 Nov 20, 2023

agithomas Nov 21, 2023

agithomas Nov 19, 2023

gpop63 commented Nov 20, 2023

gpop63 commented Nov 20, 2023

agithomas commented Nov 27, 2023

gpop63 commented Nov 27, 2023 •

edited

agithomas commented Nov 29, 2023

gpop63 commented Nov 29, 2023

agithomas left a comment

agithomas commented Nov 29, 2023

ritalwar left a comment

elasticmachine commented Nov 30, 2023

[etcd] Use prometheus metricset for etcd v3 metrics #8438

[etcd] Use prometheus metricset for etcd v3 metrics #8438

Conversation

gpop63 commented Nov 8, 2023 • edited

Overview

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Nov 8, 2023 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Nov 8, 2023 • edited

🌐 Coverage report

agithomas commented Nov 9, 2023

agithomas commented Nov 9, 2023

agithomas left a comment

Choose a reason for hiding this comment

gpop63 commented Nov 9, 2023

agithomas commented Nov 9, 2023

agithomas commented Nov 9, 2023

gpop63 commented Nov 9, 2023

agithomas Nov 19, 2023

Choose a reason for hiding this comment

gpop63 Nov 20, 2023

Choose a reason for hiding this comment

agithomas Nov 21, 2023

Choose a reason for hiding this comment

agithomas Nov 19, 2023

Choose a reason for hiding this comment

gpop63 commented Nov 20, 2023

gpop63 commented Nov 20, 2023

agithomas commented Nov 27, 2023

gpop63 commented Nov 27, 2023 • edited

agithomas commented Nov 29, 2023

gpop63 commented Nov 29, 2023

agithomas left a comment

Choose a reason for hiding this comment

agithomas commented Nov 29, 2023

ritalwar left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 30, 2023

gpop63 commented Nov 8, 2023 •

edited

elasticmachine commented Nov 8, 2023 •

edited

elasticmachine commented Nov 8, 2023 •

edited

gpop63 commented Nov 27, 2023 •

edited