-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
snmptrap -> redis JSON encoding bug #8
Comments
I'm hitting this bug as well and was trying to think about the "right" way to fix it. The problem is that some of the SNMP values are going to contain arbitrary binary strings. As far as I can tell, there's not a way to convert arbitrary binary strings to UTF-8 in a reversible way. I can see where @jordansissel is saying that adding codec support would help with the encoding, but I'm not convinced it would help for arbitrary binary strings as it would not guarantee that you can still recover the same binary on the other side of Redis, for example. So I'm thinking the best solution is to change the behavior of the Obviously, this would break compatibility with anyone using the binary strings directly, but would alleviate the problem where people expect it to 'just work' and it currently doesn't in some cases. Does this seem like a terrible idea? |
On Wed, 2016-03-16 at 13:55:28 -0700, Greg Mefford wrote:
Possible to store them in two fields? One UTF-8 'replace' encoded and
IIRC Redis can handle binaries? I'm not sure though.
How would you manipulate that from logstash without having to retort to |
I guess the other question is how to properly represent this data once it's in ElasticSearch anyway? Perhaps the real solution is to convert early (like in the input plugin) from binary MACs to say HEX or whatnot, possibly as a configurable option? |
Yeah, I think the only generic solution is to convert early in the input As far as how to process it downstream, it seems like a ruby filter is the On Thursday, March 17, 2016, NV notifications@github.com wrote:
|
(This issue was originally filed by @nvx at elastic/logstash#1636)
I have a simple logstash configuration that reads snmptraps and outputs them to redis which is json encoded.
Some messages fail to make it to redis with the logstash log reporting "Failed to convert event to JSON. Invalid UTF-8, maybe?". Looking at the code this appears to originate from within the redis output.
I have tried without specifying a codec, as well as explicitly setting the charset to BINARY. The SNMP traps do contain some non-ASCII characters (binary representations of MAC addresses and IP addresses) but they appear to be properly escaped with \xHH style notation in the output log.
The only difference I can spot between messages that make it to redis, compared ones that fail is the MAC address field. An example of a failing message has this field in the message part (Note it appears to be doubly escaped as this is from the error log which itself appears to be json encoded as well):
And the same value again as parsed by the MIBs:
Other MAC addresses that start with FC:F8:AE work, so I can only assume it is the latter half (3C:2E:18) that is breaking the encoding.
The text was updated successfully, but these errors were encountered: