Split a JSON byte stream into JSON objects/arrays. Fixes #2536

[buchgr]

Motivation:

See GitHub Issue #2536.

Modifications:

Introduce the class JsonObjectDecoder to split a JSON byte stream
into individual JSON objets/arrays.

Result:

A Netty application can now handle a byte stream where multiple JSON
documents follow eachother as opposed to only a single JSON document
per request.

@trustin @daschl @normanmaurer please review :-)

[ghost]

Build result for #2547 at dcdc69fc28ff416717c0742f13a88c399116fcb7: Failure

[ghost]

Build result for #2547 at 07ad08da328a825451e8ba3946bc27d3e4e01d83: Success

[normanmaurer]

use {@link ChannelPipeline}

[ghost]

Build result for #2547 at 99a6c15049b714610622427a11b984e82b32daea: Success

[ghost]

Build result for #2547 at 8d2755bea51a05a6ea4a4b24676ff5308587156c: Success

[ghost]

Build result for #2547 at 6d4dc4ed89eb988a7eae1d67c718df2d9ca36005: Success

[normanmaurer]

@trustin wdyt ?

[daschl]

@jakobbuchgraber I'm doing something a litte different in my project, maybe you like it:

basically using a processor to find some json markers:

    private static class MarkerProcessor implements ByteBufProcessor {

        private int marker = 0;
        private int counter = 0;
        private int depth = 0;
        private byte open = '{';
        private byte close = '}';
        private byte stringMarker = '"';
        private boolean inString = false;

        @Override
        public boolean process(byte value) throws Exception {
            counter++;
            if (value == stringMarker) {
                inString = !inString;
            }
            if (!inString && value == open) {
                depth++;
            }
            if (!inString && value == close) {
                depth--;
                if (depth == 0) {
                    marker = counter;
                }
            }
            return true;
        }

        public int marker() {
            return marker;
        }
    }

I'm doing slightly different stuff but you can use the processor for this very efficiently.
/cc @normanmaurer

[buchgr]

@daschl thanks for sharing. I started with a processor too (as I learned from @normanmaurer's FB talk :P ), but I also have to be able to look back one byte to check for escape characters (e.g. "foo " bar") and so I kinda felt the loop was nicer for my purposes.

Maybe I should benchmark both versions, but I didn't bother with these kind of micro optimizations yet 😄

[trustin]

You could also check if an exception is raised here.

[trustin]

Could you also add a test case where a string contains '{' and other potentially confusing tokens?

[trustin]

I also want to see an option that enables a streaming of a JSON array elements, whose size is potentially infinite. For example, when a client sends this:

[
"a",
{ key: "value" },
"c",
...
]

The decoder could generate a frame for each array element (i.e. ""a"", "{ key: "value" }", "c"", ...)

[ghost]

Build result for #2547 at 8250ec5554dbfbd4acd56c98cd94811fa14d449b: Failure

[buchgr]

I addressed the comments and I think it's ready for final review/merge.

[trustin]

The default maxObjectLength should be a much smaller value? 1048576?

[trustin]

Looks pretty good all in all. Left some comments for tiny things. To summarize:

• Raise a CorruptedFrameException when the received data is not JSON

[buchgr]

done

[trustin]

Nice work. Merged into 4.1 and master. 👍

[alex-vas]

It might be worth to add a comment that this implementation is only compatible with UTF-8 or ASCII encoded streams. Might event be worth it to add Charset as a constructor parameter and assert that it is UTF-8 straight away. This would make the API ready for possible future implementation of other encodings support. Same goes to LineBasedFrameDecoder and XmlFrameDecoder - both missing a comment explaining that they are expecting the stream in UTF-8.

[normanmaurer]

Are you interested in providing a PR?

…

Am 12.12.2018 um 21:04 schrieb Alex Vasiliev ***@***.***>: It might be worth to add a comment that this implementation is only compatible with UTF-8 or ASCII encoded streams. Might event be worth it to add Charset as a constructor parameter and assert that it is UTF-8 straight away. This would make the API ready for possible future implementation of other encodings support. Same goes to LineBasedFrameDecoder and XmlFrameDecoder - both missing a comment explaining that they are expecting the stream in UTF-8. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[alex-vas]

Let me give it a try. :)