Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make elasticsearch/index_summary metricset work for Stack Monitoring without xpack.enabled flag #20615

Merged
merged 18 commits into from
Nov 10, 2020
Merged

Make elasticsearch/index_summary metricset work for Stack Monitoring without xpack.enabled flag #20615

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
153b386
Bring into eventsMapping all fields in eventsMappingXpack, create ali…
sayden Aug 14, 2020
faa8ea1
Code modified. No changes in eventMapping function seemed necessary
sayden Aug 14, 2020
209462a
- Fix indentation issue
sayden Aug 14, 2020
b6e407c
- Remove aliases from non used fields in the elasticsearch mapping
sayden Aug 14, 2020
b43e1f2
Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…
sayden Aug 20, 2020
22c52f6
Reorder fields.yml?
sayden Aug 24, 2020
031d91c
Fix missing field
sayden Aug 24, 2020
78165c4
Add another missing field
sayden Aug 25, 2020
b57b32d
Updated data.json
sayden Aug 25, 2020
1993d99
Fix go integration test
sayden Aug 25, 2020
05cc97d
mage fmt update
sayden Aug 25, 2020
fd2e40b
Fix incorrect dict
sayden Aug 25, 2020
64635db
Update data.json
sayden Aug 26, 2020
b67ca49
Remove bulkstats dict from elasticsearch.go file
sayden Aug 27, 2020
35a7209
Remove data-xpack.json file
sayden Oct 8, 2020
c4934af
Fix conflict in fields.go
sayden Nov 5, 2020
2abc9ae
Remove comment
sayden Nov 5, 2020
ab25c29
Fix lint
sayden Nov 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions metricbeat/module/elasticsearch/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,3 +311,100 @@
type: boolean
description: >
Is mlockall enabled on the node?

- name: indices_stats
type: group
fields:
- name: _all
type: group
fields:
- name: primaries
type: group
fields:
- name: search
type: group
fields:
- name: query_total
type: alias
path: elasticsearch.index_summary.primaries.search.query.count
- name: query_time_in_millis
type: alias
path: elasticsearch.index_summary.primaries.search.query.time.ms
- name: bulk
type: group
fields:
- name: total_operations
type: alias
path: elasticsearch.index_summary.primaries.bulk.operations.count
- name: total_time_in_millis
type: alias
path: elasticsearch.index_summary.primaries.bulk.time.count.ms
- name: avg_time_in_millis
type: alias
path: elasticsearch.index_summary.primaries.bulk.time.avg.ms
- name: avg_size_in_bytes
type: alias
path: elasticsearch.index_summary.primaries.bulk.time.avg.bytes
- name: total_size_in_bytes
type: alias
path: elasticsearch.index_summary.primaries.bulk.size.bytes
- name: indexing
type: group
fields:
- name: index_total
type: alias
path: elasticsearch.index_summary.primaries.indexing.index.count
- name: index_time_in_millis
type: alias
path: elasticsearch.index_summary.primaries.indexing.index.time.ms
- name: is_throttled
type: alias
path: elasticsearch.index_summary.primaries.indexing.is_throttled
- name: throttle_time_in_millis
type: alias
path: elasticsearch.index_summary.primaries.indexing.throttled_time.ms
- name: total
type: group
fields:
- name: search
type: group
fields:
- name: query_total
type: alias
path: elasticsearch.index_summary.total.search.query.count
- name: query_time_in_millis
type: alias
path: elasticsearch.index_summary.total.search.query.time.ms
- name: bulk
type: group
fields:
- name: total_operations
type: alias
path: elasticsearch.index_summary.total.bulk.operations.count
- name: total_time_in_millis
type: alias
path: elasticsearch.index_summary.total.bulk.time.count.ms
- name: avg_time_in_millis
type: alias
path: elasticsearch.index_summary.total.bulk.time.avg.ms
- name: avg_size_in_bytes
type: alias
path: elasticsearch.index_summary.total.bulk.time.avg.bytes
- name: total_size_in_bytes
type: alias
path: elasticsearch.index_summary.total.bulk.size.bytes
- name: indexing
type: group
fields:
- name: index_total
type: alias
path: elasticsearch.index_summary.total.indexing.index.count
- name: index_time_in_millis
type: alias
path: elasticsearch.index_summary.total.indexing.index.time.ms
- name: is_throttled
type: alias
path: elasticsearch.index_summary.total.indexing.is_throttled
- name: throttle_time_in_millis
type: alias
path: elasticsearch.index_summary.total.indexing.throttled_time.ms
96 changes: 96 additions & 0 deletions metricbeat/module/elasticsearch/index_summary/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,54 @@
format: bytes
description: >
Total number of memory used by the segments in bytes.
- name: indexing
type: group
fields:
- name: index.count
description: TODO
type: long
- name: index.time.ms
description: TODO
type: long
- name: is_throttled
type: long
description: TODO
- name: throttle_time
type: long
description: TODO
- name: search
type: group
fields:
- name: query
type: group
fields:
- name: count
type: long
description: TODO
- name: time.ms
type: long
description: TODO
- name: bulk
type: group
fields:
- name: operations.count
type: long
description: TODO
- name: size.bytes
type: long
description: TODO
- name: time
type: group
fields:
- name: count.ms
type: long
description: TODO
- name: avg.ms
type: long
description: TODO
- name: avg.bytes
type: long
description: TODO
- name: total
type: group
fields:
Expand All @@ -54,3 +102,51 @@
format: bytes
description: >
Total number of memory used by the segments in bytes.
- name: indexing
type: group
fields:
- name: index.count
description: TODO
type: long
- name: index.time.ms
description: TODO
type: long
- name: is_throttled
type: long
description: TODO
- name: throttle_time
type: long
description: TODO
- name: search
type: group
fields:
- name: query
type: group
fields:
- name: count
type: long
description: TODO
- name: time.ms
type: long
description: TODO
- name: bulk
type: group
fields:
- name: operations.count
type: long
description: TODO
- name: size.bytes
type: long
description: TODO
- name: time
type: group
fields:
- name: count.ms
type: long
description: TODO
- name: avg.ms
type: long
description: TODO
- name: avg.bytes
type: long
description: TODO
97 changes: 89 additions & 8 deletions metricbeat/module/elasticsearch/index_summary/data.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,46 @@ var (
"bytes": c.Int("memory_in_bytes"),
},
}),
"indexing": c.Dict("indexing", s.Schema{
sayden marked this conversation as resolved.
Show resolved Hide resolved
"index": s.Object{
"count": c.Int("index_total"),
"time": s.Object{
"ms": c.Int("index_time_in_millis"),
},
},
// following field is not included in the Stack Monitoring UI mapping
sayden marked this conversation as resolved.
Show resolved Hide resolved
"is_throttled": c.Bool("is_throttled"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the field is not needed by the Stack Monitoring UI (@chrisronline may want to confirm) and it wasn't a field we were already collecting in data.go, then it's safe to not collect it at all. So I would remove this line and others like it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is correct. We need to not only collect the mappings, but also continue to collect and index any other fields we used to because we often read values from the source document that weren't considered when the finalized mappings were produced.

Copy link
Contributor

@ycombinator ycombinator Aug 14, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chrisronline So just to clarify, you're saying we should collect this field, is_throttled, but we should not include it in the fields.yml so it doesn't show up in the mappings, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood 🙂

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't really even want to index fields that aren't used and we could take another pass on which fields we read from source directly, but we're also currently in talks about potentially indexing as much data as possible (even if the stack monitoring UI doesn't use it) to enable users to build custom dashboards. Nothing is decided but I'd rather leave the functionality as is in case we do go down that path

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait a sec 😄 as far as I understood, we should map this fields in elasticsearch fields.yml mapping. But should we still map it in the metricset fields.yml file? As a Metricbeat convention, it must be mapped, AFAIK 100% of Metricbeat fields are map in fields.yml of their respective metricsets. Metricbeat is not going to fail though, we however have a commonly used python test (like this one in MySQL module https://github.com/elastic/beats/blob/master/metricbeat/module/mysql/test_mysql.py#L47) to check if all fields are map in fields.yml but fortunately it's not included in the python test code for the elasticsearch module.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.monitoring-* indices have dynamic mapping set to false, but perhaps metricbeat-* handles this differently? In that case, maybe we should add every field we are indexing to fields.yml but if the field isn't contained in the used mappings list, we set the mapping to enabled: false?

Copy link
Contributor

@ycombinator ycombinator Aug 14, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sayden I can try to help clarify about when and where to map fields for stack monitoring metricsets. 😅

Consider a field, say elasticsearch.<foo>.bar.baz that is newly (i.e. not already in master) being collected by the elasticsearch/<foo> metricset in data.go. For this field:

  1. Go ahead and define the field in the module/elasticsearch/<foo>/_meta/fields.yml file.

  2. Now, check if the field has a corresponding field in the .monitoring-es-* index mappings.

    1. If yes: let's say the corresponding field in .monitoring-es-* is some_thing.some_thing_else.qux.boom. Go ahead and define a field alias from some_thing.some_thing_else.qux.boom => elasticsearch.<foo>.bar.baz in the module/elasticsearch/_meta/fields.yml file.
    2. If no: edit the field mapping you created in step 1 in module/elasticsearch/<foo>/_meta/fields.yml and add enabled: false to it. This will ensure that the field is mapped (so the assert_fields_are_documented test assertion you are talking about in your comment will pass) but not be unnecessarily indexed.

Hopefully this now clear as mud 😃.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, @chrisronline re: enabled: false. I will edit my comment above to reflect this step.

// following field is not included in the Stack Monitoring UI mapping
"throttle_time": s.Object{
"ms": c.Int("throttle_time_in_millis"),
},
}),
// following field is not included in the Stack Monitoring UI mapping
"bulk": s.Object{
sayden marked this conversation as resolved.
Show resolved Hide resolved
"operations": s.Object{
"count": c.Int("total_operations"),
},
"time": s.Object{
"count": s.Object{
"ms": c.Int("total_time_in_millis"),
},
"avg": s.Object{
"ms": c.Int("avg_time_in_millis"),
"bytes": c.Int("avg_size_in_bytes"),
},
},
"size": s.Object{
"bytes": c.Int("total_size_in_bytes"),
},
},
"search": s.Object{
"query": s.Object{
"count": c.Int("query_total"),
"time": s.Object{
"ms": c.Int("query_time_in_millis"),
},
},
},
}),
"total": c.Dict("total", s.Schema{
"docs": c.Dict("docs", s.Schema{
Expand All @@ -64,19 +104,52 @@ var (
"bytes": c.Int("memory_in_bytes"),
},
}),
"indexing": c.Dict("indexing", s.Schema{
"index": s.Object{
"count": c.Int("index_total"),
// following field is not included in the Stack Monitoring UI mapping
"time": s.Object{
"ms": c.Int("index_time_in_millis"),
},
},
// following field is not included in the Stack Monitoring UI mapping
"is_throttled": c.Bool("is_throttled"),
// following field is not included in the Stack Monitoring UI mapping
"throttle_time": s.Object{
"ms": c.Int("throttle_time_in_millis"),
},
}),
// following field is not included in the Stack Monitoring UI mapping
"bulk": s.Object{
"operations": s.Object{
"count": c.Int("total_operations"),
},
"time": s.Object{
"count": s.Object{
"ms": c.Int("total_time_in_millis"),
},
"avg": s.Object{
"ms": c.Int("avg_time_in_millis"),
"bytes": c.Int("avg_size_in_bytes"),
},
},
"size": s.Object{
"bytes": c.Int("total_size_in_bytes"),
},
},
"search": s.Object{
"query": s.Object{
"count": c.Int("query_total"),
"time": s.Object{
"ms": c.Int("query_time_in_millis"),
},
},
},
}),
}
)

func eventMapping(r mb.ReporterV2, info elasticsearch.Info, content []byte) error {
var event mb.Event
event.RootFields = common.MapStr{}
event.RootFields.Put("service.name", elasticsearch.ModuleName)

event.ModuleFields = common.MapStr{}
event.ModuleFields.Put("cluster.name", info.ClusterName)
event.ModuleFields.Put("cluster.id", info.ClusterID)

var all struct {
Data map[string]interface{} `json:"_all"`
}
Expand All @@ -91,6 +164,14 @@ func eventMapping(r mb.ReporterV2, info elasticsearch.Info, content []byte) erro
return errors.Wrap(err, "failure applying stats schema")
}

var event mb.Event
sayden marked this conversation as resolved.
Show resolved Hide resolved
event.RootFields = common.MapStr{}
event.RootFields.Put("service.name", elasticsearch.ModuleName)

event.ModuleFields = common.MapStr{}
event.ModuleFields.Put("cluster.name", info.ClusterName)
event.ModuleFields.Put("cluster.id", info.ClusterID)

event.MetricSetFields = fields

r.Event(event)
Expand Down
Loading