Expose internal InfluxDB statistics as JSON

[auxesis]

Per #123, it would be helpful for smarter load balancers to have insight into the performance of individual InfluxDB nodes.

[nichdiekuh]

Things I'd like to see:

• total number of SELECTs,DELETEs
• RAM currently used (if possible)
• files opened total, files currently open
• connections opened total, connections currently open
• uptime

[FGRibreau]

👍 for RAM consumption & total number of SELECT/DELETE

[phobos182]

• LevelDB metrics (Bytes Compacted, Files Compacted by Level, etc..)
• Histograms on Write / Read Latency (HDRHistogram?)
• CPU Metrics (% User, etc..)

[Dieterbe]

CPU Metrics (% User, etc..)

this sounds like a job for a monitoring agent, not influx itself.

[phobos182]

@Dieterbe CPU that InfluxDB uses. IMHO it's good hygiene and practice to expose this metric so you can figure out how much resources the daemon is consuming. It is an asset when troubleshooting new configuration settings, bugs, or issues with the daemon itself.

Monitoring on the server level would just give you the entire OS. Not what you would want, or I would recommend InfluxDB would provide from it's metrics interface.

[tamsky]

Would be really really nice if instead of just "expose... as JSON", each influx node periodically wrote its own stats to influxdb as a timeseries.

As an example, KairosDB's implemention of this is at:
https://code.google.com/p/kairosdb/wiki/Statistic

[freeformz]

+1 on some "internal" series, I was thinking about this the other day.

[jvshahid]

@tamsky I think we had a github issue to do exactly that but I can't find the issue right now. May be we talked about it internally and never opened a github issue. Anyway, I agree stats should be both exposed as JSON and written to an internal timeseries.

[XANi]

@phobos182 you can easily monitor only certain process CPU/IO/Mem, using for example collectd's process plugin.

+1 for "self-writing statistics", as those stats in most cases would land in influxdb anyway.

But together with HTTP/JSON ones as it is useful for things like nagios checks, or when migrating from other monitoring systems (when you want to get stats of your new and shiny cluster)

[chobie]

How self-writing statistics works in InfluxDB cluster?

[chobie]

I'm working this issue right now. It's just json statistics api so requires some agent.

Screenshot: (running influxdb benchmark on osx)
https://www.dropbox.com/s/8bn819lq6z8283l/Screenshot%202014-06-09%2000.48.51.png

I'll send a PR when I blush up codes. probably It requires a few weeks.

[damm]

Delicious

[chobie]

for now, I've sent PR #635. It's not fully implemented yet. but it's good time to start discussion. feedback appreciated

[BarthV]

I'm using https://github.com/novaquark/sysinfo_influxdb to collec system data into influx.
It works like a charm :-) !

[BarthV]

I replied to a previous comment asking how parsing system metrics. But it's not the main topic here btw.

[ixmatus]

System statistics for each node could easily be keyed by its node hostname or id or whatever. Whether it's a series in InfluxDB or an output logfile, or a statsd output, doesn't matter to me but having metrics would be really nice to integrate with Server Density.

[XANi]

@ixmatus it still should be provided via HTTP interface. Different configurations use different data model, someone might just have one column in every metric with value, someone else might have metric per host with each different stat in different column

[schmurfy]

I am not a big fan of having influxdb write its monitoring data in an automatically created database, the way I see this the data will never be in the correct format so we will still need to query them and write them somehwere else which makes the first database useless IO.

Adding an HTTP request which returns a json blob with the data seems like the most straight forward way to do that without making any assumptions on how these data will be used.

If both are implemented the auto statistic writing feature should not be enabled by default and the http endpoint should be useable separately of this config option.

[fntlnz]

👍

[damm]

@schmurfy I think once it's exposed as a JSON endpoint it does not matter how the data is collected (or not).

Self-collecting stats is a great paradox and you should question the values it stores.

[tamsky]

InfluxDB definitely should write internal metrics to InfluxDB, and it should do so by default.

Sure, provide a configuration option to disable this, but, more importantly,
these internal metrics should be setting a very important example to the community:

"We've thought long and hard about our internal metrics, and this is how we believe they should be structured in influxdb."

If InfluxDB does not eat its own dogfood, I'm going to wonder why.

[ixmatus]

No it should not write internal metrics to InfluxDB!! Eating that kind of
dogfood is silly because you're coupling something very important too
tightly to its source.

Does every plugin author that wants to read its progam metrics data then
need to implement InfluxDB's HTTP protocol to get that data out?

An internal endpoint that just dumps out flat JSON is the simplest way to
cater to a large array of system uptime and performance monitoring tools.

On Mon, Oct 13, 2014 at 4:38 PM, tamsky notifications@github.com wrote:

InfluxDB definitely should write internal metrics to InfluxDB, and it
should do so by default.

Sure, provide a configuration option to disable this, but, more
importantly,
these internal metrics should be setting a very important example to
the community:

"We've thought long and hard about our internal metrics, and this is how
we believe they should be structured in influxdb."

If InfluxDB does not eat its own dogfood, I'm going to wonder why.

—
Reply to this email directly or view it on GitHub
#126 (comment).

Parnell Springmeyer
parnell@digitalmentat.com | digitalmentat.com | 0xDCCF89258EAD874A
http://pgp.mit.edu/pks/lookup?op=get&search=0xDCCF89258EAD874A

[tamsky]

NB:
I am not arguing against exposing internal metrics as JSON.

[schmurfy]

@damm that's what I said, we completely agree on this.
Honestly I don't care whether influxdb write stats into "itself" (since that's the idea) but I want to be able to disable it to avoid what is in my case a useless process since I won't use it.

[IngaFeick]

+1 for exposing statistics and health info via any api. Would really love to have this so that we could write a collectd plugin to monitor the db performance.

[mre]

+1

[candoom]

+1

[damm]

This issue was originally created December 30th, 2013. It's really :( to see +1's with no new contributions here.

[otoolep]

This is going to be part of the 0.9 release:

#1470
#1252

[ixmatus]

That is wonderful news.
ᐧ

On Tue, Feb 10, 2015 at 3:28 PM, otoolep notifications@github.com wrote:

This is going to be part of the 0.9 release:

#1470 #1470
#1252 #1252

—
Reply to this email directly or view it on GitHub
#126 (comment).

Parnell Springmeyer
parnell@digitalmentat.com | digitalmentat.com | 0xDCCF89258EAD874A
http://pgp.mit.edu/pks/lookup?op=get&search=0xDCCF89258EAD874A

[damm]

@otoolep would have never known as they were not referenced at all. But thank you for sharing :)

[XANi]

metrics as internal "virtual" database would be pretty elegant solution, then continous query could be used to persist wanted stats.

[beckettsean]

done in #1936