Problem parsing non-ascii letters inside json strings

[Jonatan-Nilsson]

When I run the following code

import json_stream

em_dash = '—' #chr(8212)
payload = ('{"test": "' + em_dash*10 + '"}').encode('utf-8')

chunk_size = 10

itr = (payload[i:i+chunk_size] for i in range(0, len(payload), chunk_size))

data = json_stream.load(itr)
test = data['test']

print(test)

I get this error message

OSError: I/O error while parsing (index 15): Custom { kind: Other, error: "incomplete utf-8 byte sequence from index 0" }

The problem seems to be with the rust tokenizer. If I do json_stream.load(itr, tokenizer=json_stream.tokenizer.tokenize) it works.

[smheidrich]

Regarding the bug

Thanks for reporting this! It should be fixed in json-stream-rs-tokenizer 0.4.27+. But note that if you're on macOS Linux with an aarch64 CPU, this version isn't available yet because the build broke since the last release. I'll try to fix it sometime this week.

Other remarks (performance)

By the way, as far as I can tell, passing an iterable of bytes to json-stream's load makes it pass an unseekable byte stream to the tokenizer. The default handling for this type of stream in the Rust tokenizer is really slow in order to ensure that the cursor position of the stream matches the point up to which JSON has been parsed (which for various reasons isn't ensured for seekable streams yet, even though it would be easier to do in a performant way there... quite confusing).

If you don't care about whether the stream's cursor matches the position up to which the JSON has been parsed, you can speed it up by setting the Rust tokenizer's correct_cursor parameter to False, e.g.:

from functools import partial
from json_stream_rs_tokenizer import RustTokenizer

data = json_stream.load(itr, tokenizer=partial(RustTokenizer, correct_cursor=False))

But note that this is not "officially" part of the API yet and subject to change, so to remain forwards-compatible with future releases, you could first check that this parameter is available by trying RustTokenizer(StringIO(""), correct_cursor=False) to see if it raises an exception or via "correct_cursor" in inspect.signature(RustTokenizer).parameters.

[smheidrich]

All aarch64 wheels for 0.4.27 have been uploaded now as well (they were actually broken for Linux, not macOS as I wrongly stated above).