Issue with write_http to librato

[petecheslock]

i'm using collectd version 5.4.0, and es version 0.9.x. When loading the plugin the syslog reports it's able to fire the read callback and find all the stats (per this https://gist.github.com/petecheslock/795b708d1fbddc459dc5) But as you can see in gives an error when it tries to write: Filter subsystem: Built-in target write': Dispatching value to all write plugins failed with status -1`

If I uncomment the rrdtool plugin and comment out

<target "write">
        Plugin write_http
</target>

Then the metrics get tracked but into rrd (and not up to librato obviously).
My (sanitized) collectd.conf is in the following gist.

https://gist.github.com/petecheslock/d6c47edd2e32bf35ffb9

So i'm guessing there is something about the write_http plugin that the output of this script doesnt likes, although i'm not sure where to look. Any ideas?

[phobos182]

Thanks. I will take a look.

[petecheslock]

❤️ thanks!

[josephruscio]

@mheffner could this be related to the other issue we saw with custom plugins?

[mheffner]

We noticed that with Python input plugins the write_http output will choke if the meta field is not set correctly. A ticket was filed by @joncotton for this issue: collectd/collectd#716.

As an example, we had to set the meta field as follows. I'm still unsure what form the meta parameter requires:

def reader(input_data=None):
    metric = collectd.Values();
    metric.plugin = 'customgauge'
    metric.type = 'gauge'
    metric.plugin_instance = "test_librato_metric"
    metric.values = [0.88]
    metric.meta = {'0':True}
    metric.dispatch()

[josephruscio]

/cc @obfuscurity

[joncotton]

Yup, it looks the same as my problem in Collectd with the write_http plugin. Specifically when using JSON, it trips over the meta attribute that the Python system is setting by default. (NULL would work, a non-empty dict would work, but an empty dict breaks the JSON parser.)

To narrow down: try this and look for errors. I bet it'll work.

<Plugin "write_http">
  <URL "http://example.com">
    Format "Command"  # not JSON
  </URL>
</Plugin>

The Librato API expects JSON so that's not a solution. As far as workarounds, you can assign that dummy dict to meta as @mheffner showed (implications? probably nothing if your data is delivering to Librato) or use the now-deprecated Librato writer plugin at https://github.com/librato/collectd-librato instead of write_http.

[petecheslock]

Yup - that was it. I added var.meta = {'0':True} to approx here: https://github.com/phobos182/collectd-elasticsearch/blob/master/elasticsearch.py#L222

And now the metrics are getting sent up. Thanks all. I'm not going to submit a PR cause i'm not sure if that will cause issues with non-librato usage of write_http.

[josephruscio]

@petecheslock AFAIK this isn't strictly limited to Librato. We just parse the JSON produced by write_http like any other target would. So this affects at least anyone using write_http to push JSON?

[cep21]

I've had the same issue trying to use the write_http plugin. I created a pull request at #19 Would be great to get it fixed here as well, but since it's somewhat rare I also understand if rejected.