Influxdbjson #424
Influxdbjson #424
Conversation
This was primarily intended to consume InfluxDB-style expvars,
particularly InfluxDB's `/debug/vars` endpoint.
That endpoint follows a structure like
```json
{
"httpd::8086": {
"name": "httpd",
"tags": {
"bind": ":8086"
},
"values": {
"pointsWrittenOK": 33756,
"queryReq": 19,
"queryRespBytes": 26973,
"req": 428,
"writeReq": 205,
"writeReqBytes": 3939161
}
}
}
```
There are an arbitrary number of top-level keys in the JSON response at
the configured URLs, and this plugin will iterate through all of their
values looking for objects with keys "name", "tags", and "values"
indicating a metric to be consumed by telegraf.
Running this on current master of InfluxDB, I am able to record nearly
the same information that is normally stored in the `_internal`
database; the only measurement missing from `_internal` is `runtime`,
which is present in expvars under the "memstats" key but does not follow
the expvar format and so is not consumed in this plugin.
```
$ influx -database=telegraf -execute 'SHOW FIELD KEYS FROM /expvar/'
name: expvar_influxdb_engine
----------------------------
fieldKey
blksWrite
blksWriteBytes
blksWriteBytesC
pointsWrite
pointsWriteDedupe
name: expvar_influxdb_httpd
---------------------------
fieldKey
pingReq
pointsWrittenOK
queryReq
queryRespBytes
req
writeReq
writeReqBytes
name: expvar_influxdb_shard
---------------------------
fieldKey
fieldsCreate
seriesCreate
writePointsOk
writeReq
name: expvar_influxdb_subscriber
--------------------------------
fieldKey
pointsWritten
name: expvar_influxdb_wal
-------------------------
fieldKey
autoFlush
flushDuration
idleFlush
memSize
metaFlush
pointsFlush
pointsWrite
pointsWriteReq
seriesFlush
name: expvar_influxdb_write
---------------------------
fieldKey
pointReq
pointReqLocal
req
subWriteOk
writeOk
```
|
@otoolep Does this telegraf plugin seem reasonable to you? It's directly dependent on the JSON structure of the core database's |
| } | ||
|
|
||
| // Add a tag to indicate the source of the data. | ||
| p.Tags["influxdbjson_url"] = url |
There was a problem hiding this comment.
@sparrc does adding this tag seem reasonable to you? Or do you think it would be better to allow the user to specify their own tags explicitly somehow?
There was a problem hiding this comment.
@mark-rushakoff let's change the name to just url, to be more consistent with other plugins
|
I will definitely take a look at this shortly. |
|
Yes @mark-rushakoff -- this should be fine going forward. |
|
@mark-rushakoff sorry to ask for a name change again 😅, but there aren't any other plugins that state the output format in their name, could you change to just |
|
|
||
| func (*InfluxDBJSON) SampleConfig() string { | ||
| return ` | ||
| # Reads InfluxDB-formatted JSON from given URLs. For example, |
There was a problem hiding this comment.
this should be indented by 2 spaces
There was a problem hiding this comment.
Most of this seems like it belongs more in the README, telegraf config file would be tens of thousands of lines if we had in/out examples for each plugin
|
I don't mind doing another rename and making the other requested changes, but I don't think that Maybe the underlying issue is that this plugin is attempting to consume data formatted in a way that isn't standardized per se. @otoolep said that it should be safe to rely on that format from InfluxDB, but aside from actually inspecting the endpoint (or the code in this PR), there isn't documentation on that format that I'm aware of. |
|
@mark-rushakoff but from a consistency point of view, a plain plugin name assumes that it consumes metrics from that particular service. For example, the I agree with you that it's ambiguous in general, but other collection tools do the same thing (see https://collectd.org/wiki/index.php/Plugin:MySQL) and users are accustomed to this style of naming. |
|
I wasn't familiar enough with the patterns other collection tools use. I'll rename to |
|
Had some test trouble, so rebased onto master and also squashed into a single commit (which got tests passing again). Updated PR at #449. |
(Replaces #421)
This was primarily intended to consume a JSON endpoint structed like InfluxDB's expvars endpoint under
/debug/vars.That endpoint follows a structure like
{ "httpd::8086": { "name": "httpd", "tags": { "bind": ":8086" }, "values": { "pointsWrittenOK": 33756, "queryReq": 19, "queryRespBytes": 26973, "req": 428, "writeReq": 205, "writeReqBytes": 3939161 } } }There are an arbitrary number of top-level keys in the JSON response at
the configured URLs, and this plugin will iterate through all of their
values looking for objects with keys "name", "tags", and "values"
indicating a metric to be consumed by telegraf.
Running this on current master of InfluxDB, I am able to record nearly
the same information that is normally stored in the
_internaldatabase; the only measurement missing from
_internalisruntime,which is present in the endpoint under the "memstats" key but does not follow
the point format and so is not consumed in this plugin.
(Potentially closes #366, btw.)