Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support to CEPH input. #3311

Merged
merged 26 commits into from Jan 24, 2017
Merged
Changes from 11 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
de6ae45
Initial version.
amandahla Jan 3, 2017
dd0f784
Merge remote-tracking branch 'upstream/master'
amandahla Jan 3, 2017
98715a7
Refactoring of perf.go
amandahla Jan 4, 2017
0c8ae90
Formatted data for Tag.
amandahla Jan 4, 2017
a1ff1f7
Merge remote-tracking branch 'upstream/master'
amandahla Jan 4, 2017
41c445d
First version of status metricset.
amandahla Jan 6, 2017
86982ce
Df command.
amandahla Jan 6, 2017
8868254
Command osd pool stats finished.
amandahla Jan 9, 2017
b082c45
Repo updated.
amandahla Jan 9, 2017
5f8dd37
Merge remote-tracking branch 'upstream/master'
amandahla Jan 9, 2017
208818c
Make update and config.yml corrected.
amandahla Jan 9, 2017
b8ca266
Merge with upstream.
amandahla Jan 10, 2017
1011f02
Config reading changed.
amandahla Jan 10, 2017
747debd
Config reading changed.
amandahla Jan 10, 2017
fbe6ba2
Merge remote-tracking branch 'upstream/master'
amandahla Jan 11, 2017
66bd1d4
Initial tests.
amandahla Jan 12, 2017
06fd61c
Merge remote-tracking branch 'upstream/master'
amandahla Jan 12, 2017
85a3c26
Integration tests and generated jsons.
amandahla Jan 13, 2017
9dec734
Perf metricset removed.
amandahla Jan 16, 2017
9cec164
Merge branch 'master' of https://github.com/elastic/beats
amandahla Jan 16, 2017
596fd29
Tests corrected. Fields completed.
amandahla Jan 16, 2017
46ad8db
Docker compose updated.
amandahla Jan 16, 2017
9c8e65a
New health metricset using ceph rest api.
amandahla Jan 17, 2017
48c0e38
Fields.yml generated and docker-compose.yml corrected.
amandahla Jan 18, 2017
d289374
Fields.yml generated and docker-compose.yml corrected.
amandahla Jan 18, 2017
431c99f
Fields corrected and unity test created.
amandahla Jan 19, 2017
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -90,6 +90,38 @@ metricbeat.modules:
# Password of hosts. Empty by default
#password: test123

#-------------------------------- ceph Module --------------------------------
- module: ceph
metricsets: ["perf"]
enabled: true
period: 1s

ceph:
# location of ceph binary
binary_path = "/usr/bin/ceph"
#
#####
### MetricSet perf gather statistics via the admin socket, requires socket_dir, mon_prefix, osd_prefix and socket_suffix
#####
# directory in which to look for socket files
socket_dir = "/var/run/ceph"
# prefix of MON and OSD socket files, used to determine socket type
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
# suffix used to identify socket files
socket_suffix = "asok"
#
#####
### MetricSets status,df and poolio gather statistics via ceph commands, requires ceph_user and ceph_config
#####
# Ceph user to authenticate as, ceph will search for the corresponding keyring
# e.g. client.admin.keyring in /etc/ceph
# Consult the ceph documentation for more detail on keyring generation.
user = "client.admin"
# Ceph configuration to use to locate the cluster
config_path = "/etc/ceph/ceph.conf"


#------------------------------ couchbase Module -----------------------------
#- module: couchbase
#metricsets: ["cluster", "node", "bucket"]
@@ -46,4 +46,36 @@ metricbeat.modules:
period: 10s
processes: ['.*']

#-------------------------------- ceph Module --------------------------------
- module: ceph
metricsets: ["perf"]
enabled: true
period: 1s

ceph:
# location of ceph binary
binary_path = "/usr/bin/ceph"
#
#####
### MetricSet perf gather statistics via the admin socket, requires socket_dir, mon_prefix, osd_prefix and socket_suffix
#####
# directory in which to look for socket files
socket_dir = "/var/run/ceph"
# prefix of MON and OSD socket files, used to determine socket type
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
# suffix used to identify socket files
socket_suffix = "asok"
#
#####
### MetricSets status,df and poolio gather statistics via ceph commands, requires ceph_user and ceph_config
#####
# Ceph user to authenticate as, ceph will search for the corresponding keyring
# e.g. client.admin.keyring in /etc/ceph
# Consult the ceph documentation for more detail on keyring generation.
user = "client.admin"
# Ceph configuration to use to locate the cluster
config_path = "/etc/ceph/ceph.conf"



@@ -14,6 +14,7 @@ grouped in the following categories:
* <<exported-fields-apache>>
* <<exported-fields-beat>>
* <<exported-fields-ceph>>
* <<exported-fields-cloud>>
* <<exported-fields-common>>
* <<exported-fields-couchbase>>
@@ -411,6 +412,79 @@ type: dict
Contains user configurable fields.
[[exported-fields-ceph]]
== ceph Fields
ceph Module
[float]
== ceph Fields
[float]
== df Fields
df
[float]
=== ceph.df.example
type: keyword
Example field
[float]
== osdpoolstats Fields
osdpoolstats
[float]
=== ceph.osdpoolstats.example
type: keyword
Example field
[float]
== perf Fields
perf
[float]
=== ceph.perf.example
type: keyword
Example field
[float]
== status Fields
status
[float]
=== ceph.status.example
type: keyword
Example field
[[exported-fields-cloud]]
== Cloud Provider Metadata Fields
@@ -0,0 +1,78 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-module-ceph]]
== ceph Module

This is the ceph Module.

We used code/reference from:

This comment has been minimized.

Copy link
@ruflin

ruflin Jan 17, 2017

Collaborator

If the reference is only used to get inspiration, these should be removed.

This comment has been minimized.

Copy link
@amandahla

amandahla Jan 17, 2017

Author Contributor

Since I'll change to use Rest API, this reference will be no longer necessary indeed. I'll remove.

https://github.com/influxdata/telegraf/blob/master/plugins/inputs/ceph/ceph.go
https://github.com/elastic/beats/blob/master/metricbeat/module/docker/

Thanks!


[float]
=== Example Configuration

The ceph module supports the standard configuration options that are described
in <<configuration-metricbeat>>. Here is an example configuration:

[source,yaml]
----
metricbeat.modules:
- module: ceph
metricsets: ["perf"]
enabled: true
period: 1s
ceph:
# location of ceph binary
binary_path = "/usr/bin/ceph"
#
#####
### MetricSet perf gather statistics via the admin socket, requires socket_dir, mon_prefix, osd_prefix and socket_suffix
#####
# directory in which to look for socket files
socket_dir = "/var/run/ceph"
# prefix of MON and OSD socket files, used to determine socket type
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
# suffix used to identify socket files
socket_suffix = "asok"
#
#####
### MetricSets status,df and poolio gather statistics via ceph commands, requires ceph_user and ceph_config
#####
# Ceph user to authenticate as, ceph will search for the corresponding keyring
# e.g. client.admin.keyring in /etc/ceph
# Consult the ceph documentation for more detail on keyring generation.
user = "client.admin"
# Ceph configuration to use to locate the cluster
config_path = "/etc/ceph/ceph.conf"
----

[float]
=== Metricsets

The following metricsets are available:

* <<metricbeat-metricset-ceph-df,df>>

* <<metricbeat-metricset-ceph-osdpoolstats,osdpoolstats>>

* <<metricbeat-metricset-ceph-perf,perf>>

* <<metricbeat-metricset-ceph-status,status>>

include::ceph/df.asciidoc[]

include::ceph/osdpoolstats.asciidoc[]

include::ceph/perf.asciidoc[]

include::ceph/status.asciidoc[]

@@ -0,0 +1,19 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-metricset-ceph-df]]
include::../../../module/ceph/df/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-ceph,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/ceph/df/_meta/data.json[]
----
@@ -0,0 +1,19 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-metricset-ceph-osdpoolstats]]
include::../../../module/ceph/osdpoolstats/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-ceph,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/ceph/osdpoolstats/_meta/data.json[]
----
@@ -0,0 +1,19 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-metricset-ceph-perf]]
include::../../../module/ceph/perf/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-ceph,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/ceph/perf/_meta/data.json[]
----
@@ -0,0 +1,19 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-metricset-ceph-status]]
include::../../../module/ceph/status/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-ceph,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/ceph/status/_meta/data.json[]
----
@@ -3,6 +3,7 @@ This file is generated! See scripts/docs_collector.py
////

* <<metricbeat-module-apache,Apache>>
* <<metricbeat-module-ceph,ceph>>
* <<metricbeat-module-couchbase,couchbase>>
* <<metricbeat-module-docker,Docker>>
* <<metricbeat-module-haproxy,haproxy>>
@@ -20,6 +21,7 @@ This file is generated! See scripts/docs_collector.py
--

include::modules/apache.asciidoc[]
include::modules/ceph.asciidoc[]
include::modules/couchbase.asciidoc[]
include::modules/docker.asciidoc[]
include::modules/haproxy.asciidoc[]
@@ -10,6 +10,11 @@ import (
// This list is automatically generated by `make imports`
_ "github.com/elastic/beats/metricbeat/module/apache"
_ "github.com/elastic/beats/metricbeat/module/apache/status"
_ "github.com/elastic/beats/metricbeat/module/ceph"
_ "github.com/elastic/beats/metricbeat/module/ceph/df"
_ "github.com/elastic/beats/metricbeat/module/ceph/osdpoolstats"
_ "github.com/elastic/beats/metricbeat/module/ceph/perf"
_ "github.com/elastic/beats/metricbeat/module/ceph/status"
_ "github.com/elastic/beats/metricbeat/module/couchbase"
_ "github.com/elastic/beats/metricbeat/module/couchbase/bucket"
_ "github.com/elastic/beats/metricbeat/module/couchbase/cluster"
@@ -90,6 +90,38 @@ metricbeat.modules:
# Password of hosts. Empty by default
#password: test123

#-------------------------------- ceph Module --------------------------------
- module: ceph
metricsets: ["perf"]
enabled: true
period: 1s

ceph:

This comment has been minimized.

Copy link
@ruflin

ruflin Jan 10, 2017

Collaborator

This part of the config must be moved under the module. I assume it doesn't work the way it is at the moment.

This comment has been minimized.

Copy link
@amandahla

amandahla Jan 10, 2017

Author Contributor

Moved.

# location of ceph binary
binary_path = "/usr/bin/ceph"
#
#####
### MetricSet perf gather statistics via the admin socket, requires socket_dir, mon_prefix, osd_prefix and socket_suffix
#####
# directory in which to look for socket files
socket_dir = "/var/run/ceph"
# prefix of MON and OSD socket files, used to determine socket type
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
# suffix used to identify socket files
socket_suffix = "asok"
#
#####
### MetricSets status,df and poolio gather statistics via ceph commands, requires ceph_user and ceph_config
#####
# Ceph user to authenticate as, ceph will search for the corresponding keyring
# e.g. client.admin.keyring in /etc/ceph
# Consult the ceph documentation for more detail on keyring generation.
user = "client.admin"
# Ceph configuration to use to locate the cluster
config_path = "/etc/ceph/ceph.conf"

This comment has been minimized.

Copy link
@ruflin

ruflin Jan 10, 2017

Collaborator

In general we recommend metricbeat to connect to each node and get the specific metrics for each node instead from the cluster. What is inside this file? You should use hosts: ["..."] for that if possible.

This comment has been minimized.

Copy link
@amandahla

amandahla Jan 10, 2017

Author Contributor

I used telegraf as reference and that only collects performance metrics from the MON and OSD nodes in a Ceph storage cluster. I can change the description, perhaps.

This comment has been minimized.

Copy link
@ruflin

ruflin Jan 11, 2017

Collaborator

I'm not aware of the inner workings of CEPH. Can you share some details on what MON and OSD nodes are?

This comment has been minimized.

Copy link
@amandahla

amandahla Jan 11, 2017

Author Contributor

A Ceph Storage Cluster requires at least one MON and at least 2 OSD.

A OSD (Object Storage Daemon) handles data (storage, replication, recovery...) and provide some monitoring information to MON by checking others OSD for a heartbeat.

A MON (Monitor) handles the cluster state (maps, history...).

You can have this architecture on one server/node but just for tests. Real productions environments works in a distributed way.

If you want metrics just for the daemons where metricbeat its collecting, you can use Admin Socket Stats. In our case here, it's the "perf" metricset.

If you want metrics for the whole cluster, you can use ceph commands. In our case here, it's the "status", "df" and "osdpoolstats" metricsets.

That's why I wrote those comments on config.yml. Depending on choosen metricset, you need just some parameters.

This comment has been minimized.

Copy link
@ruflin

ruflin Jan 12, 2017

Collaborator

So perf is a local stat that should be collect by all metricbeat instances and the other 3 are cluster stats, means only one of the instances connecting needs to collect them? If there are multiple MON, will they provide different data?

This comment has been minimized.

Copy link
@amandahla

amandahla Jan 12, 2017

Author Contributor

"...means only one of the instances connecting needs to collect them?" Yes.
"If there are multiple MON, will they provide different data?" No, at least it's not expected this behavior.



#------------------------------ couchbase Module -----------------------------
#- module: couchbase
#metricsets: ["cluster", "node", "bucket"]
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.