too much cpu used by hitting elasticsearch for every metric update #96

Closed
mabrek opened this Issue Mar 23, 2015 · 8 comments

Comments

Projects
None yet
5 participants
@mabrek

mabrek commented Mar 23, 2015

When I'm sending ~600 metrics every second (just a single host with collectd and 1s update interval) cyanite jvm consumes about 30% of a single core + es jvm consumes another 30%. It looks like every time metric arrives to cyanite it issues a request to es which is not really needed (same set of metrics is being reported every second).

@mabrek

This comment has been minimized.

Show comment
Hide comment
@mabrek

mabrek Mar 25, 2015

One possible solution is to maintain a set of hashes of all indexed paths in memory and check it before sending candidate path to ES.
md5 hashes of 1M paths would take only 16Mb (and some data structure overhead) but it will decrease CPU usage significantly.

mabrek commented Mar 25, 2015

One possible solution is to maintain a set of hashes of all indexed paths in memory and check it before sending candidate path to ES.
md5 hashes of 1M paths would take only 16Mb (and some data structure overhead) but it will decrease CPU usage significantly.

@EinsamHauer

This comment has been minimized.

Show comment
Hide comment
@EinsamHauer

EinsamHauer Jun 6, 2015

Anton,

16 Mb is quite underestimated in fact - there will be quite some overhead (arrays, hashtables, pointers, etc). But the idea itself is viable. In fact I became somehow desperate about these performance issues and ended up with implementing my own writer to C* (with ~ same functionality as cyanite) with expiring paths cache. Probably, mostly because I'm not clever enough to master clojure:-)

Anton,

16 Mb is quite underestimated in fact - there will be quite some overhead (arrays, hashtables, pointers, etc). But the idea itself is viable. In fact I became somehow desperate about these performance issues and ended up with implementing my own writer to C* (with ~ same functionality as cyanite) with expiring paths cache. Probably, mostly because I'm not clever enough to master clojure:-)

@jippi

This comment has been minimized.

Show comment
Hide comment
@jippi

jippi Jun 30, 2015

bump bump

jippi commented Jun 30, 2015

bump bump

@pyr

This comment has been minimized.

Show comment
Hide comment
@pyr

pyr Jun 30, 2015

Owner

Hi everyone. I expect to tackle this mid-july, sorrya bout the delay :-(

Owner

pyr commented Jun 30, 2015

Hi everyone. I expect to tackle this mid-july, sorrya bout the delay :-(

@ebpitts

This comment has been minimized.

Show comment
Hide comment

ebpitts commented Jul 7, 2015

@EinsamHauer I assume you're referring to https://github.com/EinsamHauer/disthene ?

@pyr

This comment has been minimized.

Show comment
Hide comment
@pyr

pyr Jul 7, 2015

Owner

I just started gettings things in good shape to have a nicer core with rollup capacity. Again sorry for the wait everyone.

Owner

pyr commented Jul 7, 2015

I just started gettings things in good shape to have a nicer core with rollup capacity. Again sorry for the wait everyone.

@EinsamHauer

This comment has been minimized.

Show comment
Hide comment
@EinsamHauer

EinsamHauer Jul 7, 2015

@ebpitts yes, that's what we ended up using, at least for the time being

@ebpitts yes, that's what we ended up using, at least for the time being

@pyr

This comment has been minimized.

Show comment
Hide comment
@pyr

pyr Jul 23, 2015

Owner

This problem is now addressed in #80, cheers!

Owner

pyr commented Jul 23, 2015

This problem is now addressed in #80, cheers!

@pyr pyr closed this Jul 23, 2015

@pyr pyr added this to the 0.5.1 milestone Jul 23, 2015

@pyr pyr self-assigned this Jul 23, 2015

@pyr pyr added the bug label Jul 23, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment