[Ceph] Add Integration Package with OSD Pool Stats data stream #5128

harnish-elastic · 2023-01-27T11:14:37Z

Enhancement

What does this PR do?

Added 1 data stream (OSD Pool Stats metrics).
Added data collection logic for the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
Added dashboards and visualizations.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/ceph directory.
Run the following command to run tests.
elastic-package test

Related issues

elasticmachine · 2023-01-27T11:17:58Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-02-06T13:31:40.281+0000
Duration: 16 min 45 sec

Test stats 🧪

Test	Results
Failed	0
Passed	15
Skipped	0
Total	15

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

kush-elastic · 2023-01-27T11:23:53Z

/test

elasticmachine · 2023-01-27T11:36:16Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`3/3`)	💚
Classes	100.0% (`3/3`)	💚
Methods	100.0% (`24/24`)	💚
Lines	92.547% (`149/161`)	👍 1.057
Conditionals	100.0% (`0/0`)	💚

kush-elastic

LGTM!

packages/ceph/data_stream/osd_pool_stats/fields/fields.yml

agithomas · 2023-01-30T05:55:27Z

packages/ceph/data_stream/osd_pool_stats/fields/fields.yml

+              fields:
+                - name: bytes
+                  type: long
+                  description: Number of client I/O rates write bytes per second.


Please change it to write rates

Just to clarify, Do we need to update description as Number of client I/O rates write bytes per second. -> Number of client I/O write rates bytes per second.?

Number of client I/O write rates in bytes per second

Got it. Will update wherever applicable. Thanks!

agithomas · 2023-01-30T05:56:02Z

packages/ceph/data_stream/osd_pool_stats/fields/fields.yml

+                  description: Number of client I/O rates write bytes per second.
+                  metric_type: gauge
+                  unit: byte
+                - name: operation.count


Suggest to use operation_count as there exist no more than 1 fields under operation group

We are following Elastic best practice while doing field mappings. Hence we have added count suffix wherever it's applicable. Final json structure would be like ceph.osd_pool_stats.client_io_rate.write.operation.count. Let me know if it looks good to you!

agithomas · 2023-01-30T05:57:49Z

packages/ceph/data_stream/osd_pool_stats/fields/fields.yml

+        - name: id
+          type: long
+          description: Pool ID.
+        - name: name


suggest to rename as pool_name and pool_id

agithomas · 2023-01-30T06:03:54Z

packages/ceph/data_stream/osd_pool_stats/fields/fields.yml

+            - name: read
+              type: group
+              fields:
+                - name: bytes


It may be best to change to write.byte to write.rate_per_sec , read.byte to read.rate_per_sec. This is following the convetion in oracle.sysmetric fields mapping

Based on your suggestions, final output will be ceph.osd_pool_stats.client_io_rate.write.rate_per_sec that is making rate redundant. I guess currently it is the best structure that provides the IO rates and in group only. So readability also increase as per the current json event structure. I see in oracle.sysmetric no json structure is form eventhough we can. According to me, The current best json structure for the user is ceph.osd_pool_stats.client_io_rate.write.bytes and same is applicable for the field read.bytes.

You may consider changing from ceph.osd_pool_stats.client_io_rate -> ceph.osd_pool_stats.client_io, if there are no constraints

Instead of updating ceph.osd_pool_stats.client_io_rate -> ceph.osd_pool_stats.client_io, We can update ceph.osd_pool_stats.client_io_rate -> ceph.osd_pool_stats.client_io_rate_per_sec, Because all the client_io_rate are descripted as per_sec. So the updating field mapping will be like ceph.osd_pool_stats.client_io_rate_per_sec.read.bytes. WDYT?

Go with removing operation. from everywhere. This would solve all the problems.

Every IO is an operation. So, no need to use it anywhere

Keep rest as it is.

Are you sure we should not keep the operation. in field mapping? I guess it is not creating any problems based on current structure. I/O can be operation, device, program anything. According to me its seems fine to callout operation separately.

Yes, please go ahead removing it

agithomas

LGTM

elasticmachine · 2023-02-06T14:11:40Z

Package ceph - 0.3.0 containing this change is available at https://epr.elastic.co/search?package=ceph

Add osd_pool_stats data stream

e9871b7

kush-elastic self-requested a review January 27, 2023 11:15

kush-elastic assigned harnish-elastic Jan 27, 2023

kush-elastic added enhancement New feature or request Integration:Ceph Team:Service-Integrations Label for the Service Integrations team labels Jan 27, 2023

update changelog.yml

fc9d12e

harnish-elastic marked this pull request as ready for review January 27, 2023 11:36

kush-elastic requested review from lalit-satapathy and agithomas January 27, 2023 11:37

kush-elastic approved these changes Jan 30, 2023

View reviewed changes

agithomas reviewed Jan 30, 2023

View reviewed changes

This was referenced Jan 31, 2023

[Ceph] Add Integration Package with dashboard #5152

Merged

[Ceph] Add Integration Package with OSD Performance data stream #4930

Merged

[Ceph] Add Integration Package with Cluster Disk data stream #5011

Merged

address review comments

75e3c21

This was referenced Jan 31, 2023

[Ceph] Add Integration Package with Pool Disk data stream #5070

Merged

[Ceph] Add Integration Package with Cluster Health data stream #4963

Merged

harnish-elastic requested review from agithomas and removed request for lalit-satapathy February 1, 2023 07:27

harnish-elastic mentioned this pull request Feb 2, 2023

[Ceph] Add Integration Package with OSD Tree data stream #5174

Merged

4 tasks

update README.md

88dbbfd

kush-elastic requested review from SubhrataK and lalit-satapathy February 3, 2023 05:25

agithomas approved these changes Feb 4, 2023

View reviewed changes

resolve merge conflicts

48810a6

harnish-elastic requested a review from a team as a code owner February 6, 2023 13:31

kush-elastic merged commit f273bae into elastic:main Feb 6, 2023

harnish-elastic mentioned this pull request Feb 8, 2023

[Ceph] Add Integration Package with Cluster Status data stream #5216

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ceph] Add Integration Package with OSD Pool Stats data stream #5128

[Ceph] Add Integration Package with OSD Pool Stats data stream #5128

harnish-elastic commented Jan 27, 2023 •

edited

Loading

elasticmachine commented Jan 27, 2023 •

edited

Loading

Build stats

Test stats 🧪

kush-elastic commented Jan 27, 2023

elasticmachine commented Jan 27, 2023 •

edited

Loading

kush-elastic left a comment

agithomas Jan 30, 2023

harnish-elastic Jan 30, 2023

agithomas Jan 30, 2023

harnish-elastic Jan 30, 2023 •

edited

Loading

agithomas Jan 30, 2023

harnish-elastic Jan 30, 2023

agithomas Jan 30, 2023

agithomas Jan 30, 2023

harnish-elastic Jan 30, 2023

agithomas Jan 30, 2023

harnish-elastic Jan 30, 2023

agithomas Jan 30, 2023 •

edited

Loading

agithomas Jan 30, 2023

harnish-elastic Jan 30, 2023

agithomas Jan 30, 2023 •

edited

Loading

agithomas left a comment

elasticmachine commented Feb 6, 2023

[Ceph] Add Integration Package with OSD Pool Stats data stream #5128

[Ceph] Add Integration Package with OSD Pool Stats data stream #5128

Conversation

harnish-elastic commented Jan 27, 2023 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Jan 27, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

kush-elastic commented Jan 27, 2023

elasticmachine commented Jan 27, 2023 • edited Loading

🌐 Coverage report

kush-elastic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harnish-elastic Jan 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agithomas Jan 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agithomas Jan 30, 2023 • edited Loading

Choose a reason for hiding this comment

agithomas left a comment

Choose a reason for hiding this comment

elasticmachine commented Feb 6, 2023

harnish-elastic commented Jan 27, 2023 •

edited

Loading

elasticmachine commented Jan 27, 2023 •

edited

Loading

elasticmachine commented Jan 27, 2023 •

edited

Loading

harnish-elastic Jan 30, 2023 •

edited

Loading

agithomas Jan 30, 2023 •

edited

Loading

agithomas Jan 30, 2023 •

edited

Loading