[Ceph] Add Integration Package with Cluster Health data stream

[harnish-elastic]

• Enhancement

What does this PR do?

• Added 1 data stream (Cluster Health metrics).
• Added data collection logic for the data streams.
• Added the ingest pipeline for the data streams.
• Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
• Added system test cases for the data stream.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

• Clone integrations repo.
• Install elastic-package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/ceph directory.
• Run the following command to run tests.
  elastic-package test

Related issues

• Relates Migration of Ceph Integration #4871
• Relates [Ceph] Add Integration Package with OSD Performance data stream #4930
• Relates [Ceph] Add Integration Package with Cluster Disk data stream #5011
• Relates [Ceph] Add Integration Package with Pool Disk data stream #5070
• Relates [Ceph] Add Integration Package with OSD Pool Stats data stream #5128
• Relates [Ceph] Add Integration Package with dashboard #5152
• Relates [Ceph] Add Integration Package with OSD Tree data stream #5174

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-02-06T12:45:52.918+0000

• Duration: 15 min 58 sec

Test stats 🧪

Test	Results
Failed	0
Passed	10
Skipped	0
Total	10

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[kush-elastic]

/test

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (2/2)	💚
Files	100.0% (2/2)	💚
Classes	100.0% (2/2)	💚
Methods	100.0% (16/16)	💚
Lines	91.489% (86/94)	👍
Conditionals	100.0% (0/0)	💚

[kush-elastic]

LGTM!

[agithomas]

I do not see dashboard specific files? Doesn't the datastream comes with visualisation ?

[harnish-elastic]

I do not see dashboard specific files? Doesn't the datastream comes with visualisation ?

The data stream does not have any visualization. Hence we haven't any dashboard specific files.

[agithomas]

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

[harnish-elastic]

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

We already have gone through the ceph images which you have mentioned while doing system test. Also we have gone through the https://hub.docker.com/r/ceph/daemon image.

We are facing this issue while doing docker-compose up,

We are facing this similar issues when performing ceph operations in the docker image.

Also we have referred the ceph metricbeat module how they are doing system tests.

So we have performed similar steps with our system test. But we are facing the issue related to API Secret Key.
We need to pass the API User and API Secret Key, while configuring the ceph integration. (Which we already mentioned how to get this details in our README.md)
For the API User, the environment is set in that way like we have static value for API User (demo).
But for the API Secret Key, the system is having dynamic value. (Generated by Ceph environment)
So for that we need to set the API Secret Key to environment variable and pass it while doing system test.
But somehow we are not able to do this while doing system test.

Hence we have to perform the system test using the mock server.

[agithomas]

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

We already have gone through the ceph images which you have mentioned while doing system test. Also we have gone through the https://hub.docker.com/r/ceph/daemon image.

We are facing this issue while doing docker-compose up, We are facing this similar issues when performing ceph operations in the docker image.

Also we have referred the ceph metricbeat module how they are doing system tests.

So we have performed similar steps with our system test. But we are facing the issue related to API Secret Key. We need to pass the API User and API Secret Key, while configuring the ceph integration. (Which we already mentioned how to get this details in our README.md) For the API User, the environment is set in that way like we have static value for API User (demo). But for the API Secret Key, the system is having dynamic value. (Generated by Ceph environment) So for that we need to set the API Secret Key to environment variable and pass it while doing system test. But somehow we are not able to do this while doing system test.

Hence we have to perform the system test using the mock server.

@harnish-elastic , Can you please create an issue in the elastic-package repo explaining the requirement

[harnish-elastic]

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

Please use the actual docker image for creating system testcases. Reference : https://quay.io/repository/ceph/ceph?tab=tags, https://hub.docker.com/u/ceph, https://docs.ceph.com/en/latest/install/containers/

We already have gone through the ceph images which you have mentioned while doing system test. Also we have gone through the https://hub.docker.com/r/ceph/daemon image.
We are facing this issue while doing docker-compose up, We are facing this similar issues when performing ceph operations in the docker image.
Also we have referred the ceph metricbeat module how they are doing system tests.
So we have performed similar steps with our system test. But we are facing the issue related to API Secret Key. We need to pass the API User and API Secret Key, while configuring the ceph integration. (Which we already mentioned how to get this details in our README.md) For the API User, the environment is set in that way like we have static value for API User (demo). But for the API Secret Key, the system is having dynamic value. (Generated by Ceph environment) So for that we need to set the API Secret Key to environment variable and pass it while doing system test. But somehow we are not able to do this while doing system test.
Hence we have to perform the system test using the mock server.

@harnish-elastic , Can you please create an issue in the elastic-package repo explaining the requirement

Raised issue in elastic-package repo explaining requirement. Link: elastic/elastic-package#1126

[agithomas]

metric_type mapping is missing. Please add them.

[harnish-elastic]

metric_type mapping is missing. Please add them.

Updated, thanks!

[agithomas]

Please check the comments

[agithomas]

LGTM

[elasticmachine]

Package ceph - 0.2.0 containing this change is available at https://epr.elastic.co/search?package=ceph