Metricbeat Ceph stats collection breaks for Ceph Mimic release

[ph]

This was initially reported by jrosser on discuss at https://discuss.elastic.co/t/metricbeat-ceph-stats-collection-breaks-for-ceph-mimic-release/137496

---

Metricbeat's Ceph module queries the API provided by ceph-rest-api, a command-line tool provided with Ceph versions up to Luminous. This is documented here:

http://docs.ceph.com/docs/luminous/man/8/ceph-rest-api/

This was deprecated under Luminous and has been removed completely from the latest Ceph release, Mimic. (See "Notable Changes" on this page: https://ceph.com/releases/v12-2-5-luminous-released/ )

The replacement is the more comprehensive and more secure Restful plugin for the ceph-mgr service. This is documented here:

http://docs.ceph.com/docs/mimic/mgr/restful/

This uses HTTPS and requires authentication by the client: see

https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/ceph_management_api/restful-api#enabling-and-securing-the-restful-plug-in

It would be great if MetricBeat understood the new Ceph API for collecting stats.

[alvarolobato]

Seems related to #7723

[sorantis]

Some of the stats are no longer applicable, which makes some key metrics, e.g. cluster health unusable. cc: @andresrc, @exekias

[eihkoh]

Is there any chance that Metricbeat Ceph module will support the new Ceph Restful API?
Since Ceph mimic release the ceph-rest-api is changed to Ceph Restful API as already mention above.

[sorantis]

@eihkoh Updated Ceph module is coming in the next release.

[eihkoh]

@eihkoh Updated Ceph module is coming in the next release.

@sorantis.. Thanks a lot. which release will it be? 7.6.x?

[sorantis]

@eihkoh 7.7.0

[wyllys66]

Does it use the correct API base url? The code I see uses "/api/v0.1", which is incorrect for ceph Mimic and later releases.

The code in the 7.7 branch as of today is still broken and using the old API.
For example:

beats/metricbeat/module/ceph/osd_df/osd_df.go

Line 30 in bda332d

defaultPath = "/api/v0.1/osd/df"

[mtojek]

@wyllys66 It's not broken, it uses old API for support compatibility with old releases. As you noticed it doesn't work with Mimic and later releases. Newer releases base on Ceph Manager Daemon and the Restful module (https://docs.ceph.com/docs/master/mgr/restful/). Metricsets with names starting with "mgr_" prefix are compatible with ceph-mgr.

For reference: https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-ceph.html

[wyllys66]

OK, thanks for the clarification.

[mtojek]

I will resolve this one in favor of #7723

[lihengxuan]

I use metricbeat 7.6.2 collect ceph data(mimic ) ,the conig file is:

• module: ceph
  metricsets:
  • mgr_cluster_disk
  • mgr_osd_perf
  • mgr_pool_disk
  • mgr_osd_pool_stats
  • mgr_osd_tree
    period: 1m
    hosts: [ "https://localhost:8003" ]
    username: "metricbeat"
    password: "8bd7d8d4-7092-46eb-9146-e5086b0e8b6d"
    ssl:
    verification_mode: none
    but I can't collect data,log file is
    2021-11-04T17:32:25.347+0800 ERROR instance/beat.go:933 Exiting: 1 error: 5 errors: metricset 'ceph/mgr_cluster_disk' not found; metricset 'ceph/mgr_osd_perf' not found; metricset 'ceph/mgr_pool_disk' not found; metricset 'ceph/mgr_osd_pool_stats' not found; metricset 'ceph/mgr_osd_tree' not found
    I can understand why I failed.