-
Notifications
You must be signed in to change notification settings - Fork 375
Corrupted messages are lost #65
Comments
Bummer that we are delivering invalid UTF-8 data. I'd love to see the invalid data too so that we can fix things on our side. Couple options here:
I think the simplest possible thing is patching DelimitedStringReader, but it also doesn't give much flexibility. I suspect its good enough for this though. |
I was mistaken about the data in the firehose messages being invalid UTF-8. The data was valid UTF-8 just not valid JSON and the cause was traced back to multibyte UTF-8 mangling. This occurred back around March 12-14 2013. I communicated with Arne and it was fixed. To quote Arne:
In any case, I would like to cover both possibilities (invalid UTF-8 and invalid JSON). I'm fairly new to Java so I might have overlooked something, but it is my understanding that the String constructors do not throw on invalid data. Instead they silently replace it. This is why I ended up at CharsetDecoder. I've written this JUnit test to demonstrate the problem:
I realize this is a bit pedantic and if you think it is out of the scope of hbc to check for encoding errors, I'd agree and just have my own HosebirdMessageProcessor. |
Ahhh didn't know that bit about the String constructor replacing bad data. So, my take is that this was a bug on our end that we don't expect to regress again and I'd rather leave this bit simple in our implementation. |
Sounds good to me. |
My firehose client I am trying to replace with hbc has a feature that saves messages which aren't valid UTF-8 to files. This has proven useful because it has found and helped fix bugs in the Firehose. Right now the utf8 decoding happens roughly with this stack:
com.twitter.hbc.common.DelimitedStreamReader.readLine
...
com.twitter.hbc.core.processor.StringDelimitedProcessor.processNextMessage
com.twitter.hbc.httpclient.ClientBase.processConnectionData
readLine() calls new String(...), constructor which does this with invalid utf8:
I believe this means the original data will be lost and worse the problem will likely be hidden. It would be nice if we could get at the invalid data to log it. One option would be returning byte[] instead of String. Another option would be to use a CharsetDecoder with CodingErrorAction.REPORT and perhaps add an event with the bad byte[]. The second option is a less invasive API change.
Thoughts?
The text was updated successfully, but these errors were encountered: