feat(encoding): add json/stream.ts

[ayame113]

close #2208

• JSONLinesParseStream parses JSON lines, NDJSON and JSON Text Sequences.
• JSONLinesStringifyStream stringifies JSON lines, NDJSON and JSON Text Sequences.
• ConcatenatedJSONParseStream parses Concatenated JSON.
• ConcatenatedJSONStringifyStream stringifies Concatenated JSON.

Implementations have been ported from https://github.com/ayame113/jsonlines/tree/9e969323a0b09d573a4a35f0d466866bb62e1a67.

#2227 needs to be merged first. The CI should turn green when #2227 is merged into main.

[kt3k]

@ayame113 LGTM. Great!

[bartlomieju]

LGTM, too!

[crowlKats]

LGTM, besides two minor nitpicks

[lucacasonato]

TBH, I think these combinators are too specific. A JSONLinesParseStream is the same as a LineStream + JSONParseStream. For stream combinators we should provide low level primitives that users can orchestrate together themselves instead of providing one trick ponys.

I think JSONParseStream (that just parses each chunk as is), and JSONStringifyStream (which stringifies each chunk as is), are much better combinators.

[lucacasonato]

For concatenation, I think a stream combinator that has an API in the form JoinStream<T>((i: number) => T | Promise<T>) that calls the callback for each chunk, and enqueue the result after each chunk. Please bikeshed the name though, as JoinStream is pretty terrible.

[nayeemrmn]

IMO we should change this and the jsonc module to use Json casing as we do with fromFileUrl()

[kt3k]

@lucacasonato

TBH, I think these combinators are too specific. A JSONLinesParseStream is the same as a LineStream + JSONParseStream. For stream combinators we should provide low level primitives that users can orchestrate together themselves instead of providing one trick ponys.

This suggestion makes sense to me. You mean the users should be able to do:

const readable = body!
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new TextDelimiterStream(separator))
  .pipeThrough(new JSONParseStream());

instead of:

const readable = body!
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new JSONLinesParseStream()); 

I also also agree with reducing JSONLinesStringifyStream and ConcatenatedJSONStringifyStream to single JSONStringifyStream. These two look almost the same.

For concatenation, I think a stream combinator that has an API in the form JoinStream((i: number) => T | Promise) that calls the callback for each chunk, and enqueue the result after each chunk. Please bikeshed the name though, as JoinStream is pretty terrible.

Is this suggestion about ConcatenatedJSONParseStream? I don't see how JoinStream covers what ConcatenatedJSONParseStream does. In my view ConcatenatedJSONParseStream is not trivial thing and can't be reduced to lower level primitives.

[kt3k]

IMO we should change this and the jsonc module to use Json casing as we do with fromFileUrl()

but we already have JSON. Json casing feels strange to me..

[kt3k]

wait we also have URL..

[ayame113]

Maybe the usage will be like this?

// parses JSON lines, NDJSON and JSON Text Sequences
const readable = body!
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new TextDelimiterStream(separator)) // or TextLineStream
  .pipeThrough(new JSONParseStream());

// parses Concatenated JSON
const readable = body!
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new ConcatenatedJSONParseStream());

// stringifies JSON lines, NDJSON, JSON Text Sequences and Concatenated JSON
readable
  .pipeThrough(new JSONStringifyStream({ prefix, suffix })) // default of prefix is "", default of suffix is "\n"
  .pipeThrough(new TextEncoderStream());

// or
// readable
//  .pipeThrough(new JSONStringifyStream())
//  .pipeThrough(new MappingStream((str) => `${prefix}${str}${suffix}`)) // The name is tentative
//  .pipeThrough(new TextEncoderStream());

I don't have time this week, so I think I'll be working on the weekend.

[kt3k]

@ayame113 The interface looks cleaner to me. Let's switch to it

[ayame113]

The current interface looks like this:

• new JSONParseStream() - Parse each chunk as JSON
• new ConcatenatedJSONParseStream() - Parse Concatenated JSON
• new JSONStringifyStream({prefix, suffix}) - Convert each chunk to a JSON string and add a prefix and suffix

---

What about JSON casing? Do I need to change it?

[bartlomieju]

I'm also in favor of keeping JSON casing.

[kt3k]

LGTM again.

I'm in favor of JSON casing.