add a heka message decoder config that doesn't clobber the message metadata #79
Conversation
…tadata (default_headers)
relud
changed the title
|
This decoder is for processing messages that conform to the Heka JSON schema. It appears this is some derivation of that schema and should really be its own cloudops decoder. |
|
I'm fine making a separate decoder, but the schema of the logs is the same, the difference is where I put it in the heka message, in order to preserve |
|
It would be nice to be able to have access to |
| local M = {} | ||
| setfenv(1, M) -- Remove external access to contain everything in the module | ||
|
|
||
| function decode(data, dh) | ||
| local msg = cjson.decode(data) | ||
| if cfg.preserve_metadata then | ||
| if type(msg.Fields) == "table" then | ||
| for k,v in pairs(msg.Fields.Fields) do |
There was a problem hiding this comment.
I need to see your original input as this implies it does not conform to the Heka message schema as there is no nested Fields of Fields hash.
There was a problem hiding this comment.
That is a typo, it is supposed to be msg.Fields
|
| local M = {} | ||
| setfenv(1, M) -- Remove external access to contain everything in the module | ||
|
|
||
| function decode(data, dh) | ||
| local msg = cjson.decode(data) | ||
| if cfg.preserve_metadata then |
There was a problem hiding this comment.
I don't understand what this comment means as you have access to everything in the current message and the current sandbox environment.
dh.Type and msg.Type may have different values here.
With the previous behavior there was no way to preserve dh.Type so that it could be used in a message matcher on an output.
With the new behavior proposed here, setting cfg.preserve_metadata would allow them to be referenced in a message matcher as Type and Fields[Type] respectively, with the caveat that msg.Fields.agent (or similar) would have to be referenced in a message matcher as Fields[Fields.agent]
|
closing this in favor of a new decoder |
There is useful metadata about a message that is stored in default_headers, which is clobbered by the default heka json decoder. For the cloudops' logging pipeline it's important to preserve those fields, because they are required in order to properly route messages.