Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Ceph] Add Integration Package with Pool Disk data stream #5070

Merged
merged 6 commits into from
Feb 8, 2023
Merged

[Ceph] Add Integration Package with Pool Disk data stream #5070

merged 6 commits into from
Feb 8, 2023

Conversation

harnish-elastic
Copy link
Contributor

@harnish-elastic harnish-elastic commented Jan 20, 2023 •
edited
Loading

  • Enhancement

What does this PR do?

  • Added 1 data stream (Pool Disk metrics).
  • Added data collection logic for the data streams.
  • Added the ingest pipeline for the data streams.
  • Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
  • Added system test cases for the data stream.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  • Clone integrations repo.
  • Install elastic-package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/ceph directory.
  • Run the following command to run tests.
    elastic-package test

Related issues

@elasticmachine
Copy link

elasticmachine commented Jan 20, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-02-07T13:13:42.055+0000

  • Duration: 16 min 7 sec

Test stats 🧪

Test Results
Failed 0
Passed 26
Skipped 0
Total 26

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@kush-elastic kush-elastic self-requested a review January 20, 2023 12:39
@kush-elastic kush-elastic added enhancement New feature or request Integration:Ceph Team:Service-Integrations Label for the Service Integrations team labels Jan 20, 2023
@kush-elastic
Copy link
Collaborator

/test

@kush-elastic kush-elastic marked this pull request as ready for review January 20, 2023 12:42
@elasticmachine
Copy link

elasticmachine commented Jan 20, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (5/5) 💚
Files 100.0% (5/5) 💚 6.25
Classes 100.0% (5/5) 💚 6.25
Methods 100.0% (41/41) 💚 13.869
Lines 92.57% (299/323) 👍 6.644
Conditionals 100.0% (0/0) 💚

kush-elastic
kush-elastic approved these changes Jan 24, 2023
View reviewed changes
Copy link
Collaborator

@kush-elastic kush-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@harnish-elastic harnish-elastic mentioned this pull request Jan 27, 2023
Merged
4 tasks
agithomas
agithomas reviewed Jan 30, 2023
View reviewed changes
Copy link
Contributor

@agithomas agithomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{\"stored\":2142673,\"objects\":4,\"kb_used\":6336,\"bytes_used\":6488064,\"percent_used\":8.4362458437681198e-05,\"max_avail\":25633505280}},

can we have these fields recorded and mapped ?

packages/ceph/data_stream/pool_disk/fields/fields.yml Outdated
description: Available bytes of the pool.
metric_type: gauge
unit: byte
- name: id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please change to pool_id

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this data stream, we have pool_disk as parent field. So the current output will be like ceph.pool_disk.id. Based on the suggested scenario, we will have ceph.pool_disk.pool_id. So I think this change would make pool redundant. WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please keept is pool_id

packages/ceph/data_stream/pool_disk/fields/fields.yml Outdated
- name: id
type: long
description: Id of the pool.
- name: name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please change to pool_name

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this data stream, we have pool_disk as parent field. So the current output will be like ceph.pool_disk.name. Based on the suggested scenario, we will have ceph.pool_disk.pool_name. So I think this change would make pool redundant. WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please keep it as pool_name

@harnish-elastic
Copy link
Contributor Author

{\"stored\":2142673,\"objects\":4,\"kb_used\":6336,\"bytes_used\":6488064,\"percent_used\":8.4362458437681198e-05,\"max_avail\":25633505280}},

can we have these fields recorded and mapped ?

Based on the response we have ingested objects, bytes_used and max_avail fields. For the kb_used and bytes_used both fields are having redundant data. Hence we can drop kb_used. We will include stored and percent_used fields in data colection.

This was referenced Jan 31, 2023
Merged
Merged
Merged
Merged
Merged
@harnish-elastic harnish-elastic requested review from agithomas and removed request for lalit-satapathy and SubhrataK February 3, 2023 05:22
@harnish-elastic harnish-elastic requested a review from a team as a code owner February 7, 2023 13:13
agithomas
agithomas approved these changes Feb 7, 2023
View reviewed changes
Copy link
Contributor

@agithomas agithomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kush-elastic kush-elastic merged commit 1784d35 into elastic:main Feb 8, 2023
@elasticmachine
Copy link

Package ceph - 0.5.0 containing this change is available at https://epr.elastic.co/search?package=ceph

@harnish-elastic harnish-elastic mentioned this pull request Feb 8, 2023
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kush-elastic kush-elastic kush-elastic approved these changes

@agithomas agithomas agithomas approved these changes

@lalit-satapathy lalit-satapathy Awaiting requested review from lalit-satapathy

@SubhrataK SubhrataK Awaiting requested review from SubhrataK

Assignees

@harnish-elastic harnish-elastic

Labels
enhancement New feature or request Integration:Ceph Team:Service-Integrations Label for the Service Integrations team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@harnish-elastic @elasticmachine @kush-elastic @agithomas