Data Pipeline Format

[crberube]

Right now there are a couple of quirks with the way that result data in formatted through the data pipeline. The biggest issue for us right now is the use of MarshalIdent on the JSON. I see that it was used in the past for some sort of Kafka Connector, but the result leaves us with JSON where each element begins on a new line with a prefix. This breaks a lot of data processing applications that expect JSON elements to be newline delimited.

A second quirk I noticed is that the JSON message frame saves the payload as a string of escaped JSON. I suspect this has something to do with the encryption option for Kafka? However, when things are not encrypted, it would be much cleaner if the all of the JSON was marshaled at once such that the payload field was simply another level of the overall JSON object as opposed to a string containing the JSON data.

I realize that these changes would amount to breaking changes in the data generated by the data pipeline, but would it be reasonable to add some sort of configuration option(s) to allow for these changes? This is something that I could address ASAP if we can figure out a way to cleanly handle this.

[zhouzhuojie]

I think it's safe to change from MarshalIndent to Marshal, no blockers.

For JSON message frame, we can add a field, for example, payload_raw_json with type json.RawMessage, and then env configuration to pick which field to use, what do you think?

[zhouzhuojie]

Related to #203

[crberube]

Sounds like a good solution, thanks! I will get a PR put together.

[zhouzhuojie]

I’m also working on a refactoring PR including this issue now, if you have time, I can use your review.

[zhouzhuojie]

@crberube #233

[crberube]

Awesome, I will take a look.