Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory usage of elasticsearch/index metricset #16503

Closed
ycombinator opened this issue Feb 21, 2020 · 3 comments · Fixed by #16538
Closed

Reduce memory usage of elasticsearch/index metricset #16503

ycombinator opened this issue Feb 21, 2020 · 3 comments · Fixed by #16538
Assignees
Labels
bug Feature:Stack Monitoring Metricbeat Metricbeat module Team:Services (Deprecated) Label for the former Integrations-Services team

Comments

@ycombinator
Copy link
Contributor

The elastiscearch/index metricset (when xpack.enabled: true is set) consumes memory proportional to the # of indices in the ES cluster and the size of the cluster state, specifically the sizes of the GET _stats and GET _cluster/state responses.

This is somewhat expected as the metricset needs data from those two API calls to create type: index_stats documents in .monitoring-es-* indices.

However, it may be possible to reduce the memory consumed by this metricset's code. Concretely, it would be worth looking into exactly which fields from the API responses are being used (and if the rest could be excluded) and also whether switching to a streaming JSON parser might help.

@ycombinator ycombinator added module Metricbeat Metricbeat Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team labels Feb 21, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/stack-monitoring (Stack monitoring)

@ycombinator ycombinator self-assigned this Feb 21, 2020
@ycombinator
Copy link
Contributor Author

ycombinator commented Feb 24, 2020

@jsoriano kindly let me pick his brain on this over Zoom. Summarizing our conversation:

  • This is not a common problem with other metricsets but we have encountered it before with the prometheus module.
  • The raw API response sizes are not massive but it's the parsed objects that are. So switching to a streaming JSON parser is probably the way to go here.

Thanks @jsoriano for chatting about this and validating some of my thinking! ❤️

@holygrolli
Copy link

@ycombinator We still encounter Metricbeat being OOMKilled. After rolling out Metricbeat 7.6.2 we experienced lower memory consumption, because we could run Metricbeat with 500Mi pod limit (before we needed much more). Now after two weeks we have more indices and shards (between 300 and 400 shards) and now we have again OOMKilled our Metricbeat running on the K8s node of the current master.
You mentioned 1.79 MB memory usage in #16538 so still having 500 Mi pod limit seems to be very much.
Perhaps you can give some insights if the number of shards might be too much compared to your memory usage statement. The Metricbeat daemonset now consumes unnecessary much memory limit on a node.
In the end I will open a new support case?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Feature:Stack Monitoring Metricbeat Metricbeat module Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants