This repository was archived by the owner on Nov 21, 2023. It is now read-only.
Bug 1447851 - Add content decoding for landfill #209
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Documents from landfill will be decoded directly into their string representation. The logic for _parse_heka_record is generally unnecessary because fields are not extracted when dumped to landfill.
whd
approved these changes
May 3, 2018
Member
whd
left a comment
whd
left a comment
There was a problem hiding this comment.
LGTM, the only weird case I could see if when Fields[content] has a non-gzip non-utf8 payload, which we might want to throw an except UnicodeDecodeError on as we do when decoding normal fields.
Contributor
Author
|
@whd I caught a bug after writing the test for an invalid utf-8 string. Is throwing a |
Member
|
This is indeed the "weird case" I mentioned in my previous comment; my intent was to suggest that we do what we already do when decoding normal fields (effectively, drop the field). |
Contributor
Author
|
Merging this in, I had been waiting for Python 3 support, but it may be better to rebase on this new functionality. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
bug link
Option b seemed like a good choice. Landfill messages will be parsed directly into a string, which can be later decoded using
json.loadsif needed.