Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming format #2

Closed
rmarx opened this issue Apr 4, 2019 · 0 comments
Closed

Streaming format #2

rmarx opened this issue Apr 4, 2019 · 0 comments

Comments

@rmarx
Copy link
Contributor

rmarx commented Apr 4, 2019

Plain JSON is not entirely streamable... it requires its fields to be closed properly (by ] and }) at the end.

e.g.,

"events": [
           { "key": "val" }

Will fail, but

"events": [
           { "key": "val" }
]

will succeed.

However, one could employ a streaming JSON parser (e.g., http://oboejs.com/) and ignore unclosed fields at the end that way.
For the way the format is currently defined, an implementation would then write the "header" of the qlog file, and then it could stream individual events that are just appended to the file.
If the file isn't properly closed, that's not a problem: the streaming parser user just ignores those errors.
However, this breaks compatibility with the built-in parsers in many stdlibs and the browsers themselves.
It would also be possible to write a simple postprocessing tool that simply appends the necessary "closing" characters to a qlog file if it isn't complete, though that adds another step in a pipeline...

There are also various JSON-subformats that address this problem (see https://en.wikipedia.org/wiki/JSON_streaming), but they also do not seem to be supported in the standard JSON parsers for many platforms...

Given all this, my personal preference is to stay with just the normal JSON format and tools are recommended to use a streaming parser.

Example of how the browser's built-in JSON parser does not support special JSON:
2019-04-04 17_02_41

Example of how a streaming parser (oboe) does handle this properly:
proof

rmarx pushed a commit that referenced this issue Aug 31, 2020
- Removes the event_fields optimization. Relates to #30. Fixes #89.

- Removes several points of complexity wrt the group_id field, as they were not being used in practice.

- Makes JSON the default serialization option. Fixes #101.

- Adds a proper streaming option with NDJSON. Relates to #106. Fixes #109, #2.

- Generally tightens the text and adds more examples.
@rmarx rmarx closed this as completed Sep 5, 2020
auliasadil pushed a commit to auliasadil/qlog-jarkom that referenced this issue May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant