New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

write_http JSON parsing of Python plugins fails on meta attribute #716

Closed
joncotton opened this Issue Aug 25, 2014 · 5 comments

Comments

Projects
None yet
3 participants
@joncotton

joncotton commented Aug 25, 2014

I have a number of custom Python plugins that output data fine with the CSV writer and with write_http using Format "Command". I'm using Collectd 5.4.1 in debug mode. When I change the formatting to JSON, the write_http plugin breaks with:

plugin: plugin_write: Writing values via write_http.
format_json: values_to_json: buffer = [100];
format_json: dstypes_to_json: buffer = ["gauge"];
format_json: dsnames_to_json: buffer = ["value"];
Filter subsystem: Built-in target `write': Dispatching value to all write plugins failed with status -1.

If this meta_data_to_json() block of code doesn't run, i.e. I comment it out, then the plugin runs. Data is written and, when the 4K buffer is full, sent over the wire. For this testing I'm using netcat on localhost and I see the data POSTed and collectd syslog log entries like:

format_json: dsnames_to_json: buffer = ["value"];
format_json: value_list_to_json: buffer = {......}
write_http plugin: localhost:8080 buffer 735/4096 (17.9443%)
...
write_http plugin: wh_flush_nolock: timeout = 0.000; send_buffer_fill = 3920;

In utils_format_json.c, vl->meta is not NULL at line 363 and inside the function at line 250, keys_num is zero, which triggers this if condition that returns an error. I'm not setting any metadata—didn't even know it was a feature—so the problem might be with the way the Python plugin system sets up the initial meta attribute. Or it might be simply that the JSON parser can't handle an empty dictionary (e.g. {}).

Here's a hack and a distilled sample of Python plugin code:

import collectd
val = collectd.Values(
    plugin="test",
    type="gauge",
    type_instance="metric1",
    meta={'0': True},  # HACK with this dummy dict in place JSON parsing works
    value=100)
val.dispatch()
@octo

This comment has been minimized.

Show comment
Hide comment
@octo

octo Sep 2, 2014

Member

It appears that the python plugin unconditionally creates a meta_data_t* in src/pyvalues.c:cpy_build_meta, which may be empty. It shouldn't do this, but neither should the JSON formatter create invalid JSON in that case. I'll take care of this.

Member

octo commented Sep 2, 2014

It appears that the python plugin unconditionally creates a meta_data_t* in src/pyvalues.c:cpy_build_meta, which may be empty. It shouldn't do this, but neither should the JSON formatter create invalid JSON in that case. I'll take care of this.

octo added a commit that referenced this issue Sep 2, 2014

@octo

This comment has been minimized.

Show comment
Hide comment
@octo

octo Sep 2, 2014

Member

Meh, 7acb8df is not properly referenced :( Anyway, either commit should fix the issue, I've done both for completeness sake. They're in collectd-5.3 for now and will eventually be merged into collectd-5.4 and master.

Member

octo commented Sep 2, 2014

Meh, 7acb8df is not properly referenced :( Anyway, either commit should fix the issue, I've done both for completeness sake. They're in collectd-5.3 for now and will eventually be merged into collectd-5.4 and master.

@octo octo closed this Sep 2, 2014

@obfuscurity

This comment has been minimized.

Show comment
Hide comment
@obfuscurity

obfuscurity Sep 2, 2014

❤️ 🍻

obfuscurity commented Sep 2, 2014

❤️ 🍻

@joncotton

This comment has been minimized.

Show comment
Hide comment
@joncotton

joncotton Sep 2, 2014

Hurray! Thanks.

joncotton commented Sep 2, 2014

Hurray! Thanks.

@octo

This comment has been minimized.

Show comment
Hide comment
@octo

octo Sep 2, 2014

Member

@joncotton thank you for your detailed bug report :-)

Member

octo commented Sep 2, 2014

@joncotton thank you for your detailed bug report :-)

cep21 added a commit to cep21/collectd-elasticsearch that referenced this issue Dec 3, 2014

cep21 added a commit to signalfx/collectd-elasticsearch that referenced this issue Dec 15, 2014

stewmmf pushed a commit to stewmmf/collectd-rabbitmq that referenced this issue Mar 12, 2015

kportertx added a commit to aerospike/aerospike-collectd that referenced this issue May 9, 2015

dbravender added a commit to dbravender/collectd-mesos that referenced this issue Jul 7, 2015

rayrod2030 pushed a commit to rayrod2030/collectd-mesos that referenced this issue Jul 7, 2015

dbravender added a commit to dbravender/collectd-marathon that referenced this issue Jul 8, 2015

klynch added a commit to klynch/collectd-marathon that referenced this issue Jul 10, 2015

ryanvm added a commit to ryanvm/collectd-rabbitmq that referenced this issue Oct 16, 2015

Hack to support write_http plugin in Collectd <5.5
There's a bug with Collectd's write_http plugin that is present in
Collectd 5.4.1 (the version distributed with Ubuntu LTS). The result
is that RabbitMQ stats cannot be pushed as JSON with the write_http
plugin. This affects people publishing stats to Librato.

For more detail, see the following bug reports:

phobos182/collectd-elasticsearch#15

collectd/collectd#716

sirmax123 added a commit to sirmax123/collectd-iostat-python that referenced this issue Mar 17, 2016

deniszh added a commit to deniszh/collectd-iostat-python that referenced this issue Mar 18, 2016

indpnday added a commit to conekia/redis-collectd-plugin that referenced this issue Apr 7, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment