Streaming multi-byte UTF8 characters not being parsed correctly

[jlank]

Hey @dscape! I was using another streaming parser (jsonparse) when I found this class of bug, and checked clarinet to see if it was present, turns out it is. To save github some bytes of storage I'll link to the bug description at creationix/jsonparse.

In a nutshell, when you do .toString() on a streamed buffer, if the stream breaks between a multi-byte utf8 character, it will not be able to properly convert the split character, and it ends up putting two replacement characters in the stream instead. I haven't devised a specific solution for clarinet yet, but I know what needs to happen.

I also wrote a little demo repo so you can see the bug in action.

guess I didn't save those storage bytes after all.

[jlank]

Oh also if you are aware of any other semi-popular streaming json parsers I'd be down to check those out for the same issue. I did a search on npm but came out of it with basically just clarinet and jsonparse as the big ones being used.

[dscape]

@jlank can you please reopen this with a pull request?

Thank you!