Streams are hot, FileReader.readAs____ is not

[jimmywarting]

Large Blob/Files becomes an issue if you don't slice it into chunks...

Streams brings other cool feature like pipe-ing, back pressure & transforming
Currently I'm using Response (part of Fetch) to transform a blob to a stream that i can "read into" step by step

var stream = new Response(blob).body
var reader = stream.getReader()

This works okey. but it feels hacky.
Could we get new method to get a stream from File & Blob's please?

[domenic]

This would be a great addition. When I asked people for how they would use streaming uploads in the browser, many of them had to do with getting a file from <input type="file">, modifying it in a streaming fashion (e.g. validating or prepending a header, or interleaving two uploaded files) and then sending them on to a server. The ability to turn a Blob/File into a ReadableStream is crucial for this.

I'd suggest blob.stream(), as a counterpart to the promise-returning blob.arrayBuffer() and blob.text() that were mentioned in #39 (comment).

[jimmywarting]

Just thought about something. If we have .arrayBuffer() and .text() why not just throw in .json() as well? ✌️
The fetch api has it, so why shouldn't blob's?

Otherwise you end up doing the same thing as (new Response(Blob)).json() which is the same kind of hack. As a bonus it rejects invalid json. Just an idea

[inexorabletash]

Blob.toStream() (the concept, not the name) was brought up at TPAC F2F and everyone in the room who had an opinion seemed to be a fan.

Also implies we should do FormData.toStream() (again, the idea not the name) for symmetry with other BodyInit types.

Open question raised in the discussion: do we need to compute the total size synchronously to be able to report progress and populate Content-Length headers? Is that doable, i.e. is the encoded size + separators knowable given data length? Is this compatible with current architectures (i.e. Chrome's browser/renderer process split)?

@annevk - did I miss anything?

[annevk]

That sounds about right. I think for FormData we concluded that exposing the size would be too hard (due to various encodings being in play). (We could also consider URLSearchParams I suppose, although that stringifies so maybe not.)

I guess the other question is whether there is some ToStream operation that takes an arbitrary object and invokes toStream() on it. Whether there's a protocol in play or just some consistency we try to enforce. (And if there's a protocol in play we might not want to use a method invocation to let browsers more easily make certain optimizations.)

As for Content-Length, we're unsure how that can work in general for stream uploads (and not needed in H2 aiui) since we need to take into account that the developer might be malicious. It might be best to simply not support Content-Length use cases since it's going away anyway.

[domenic]

Regarding size, I think that is a somewhat orthogonal issue that can be solved separately. We could add .stream() ASAP, and later add .size if possible, or figure out some out-of-band way of passing through the size specific to whatever progression framework we come up with, or... For now, no Content-Length would get set.

I don't quite remember what purpose the to-stream-able protocol would serve. I guess maybe some internal protocol that produces a { readableStream, optionalLength } structure could be used to remove the switch statements in Fetch, but that's just spec-factoring.

@inexorabletash, I can prioritize working on this next week if you think the spec is ready to accept a PR. We're still figuring out how to interface streams with other specs nicely, so I think having me do the initial work for this spec would be nice. So, a few concrete questions:

• Is this spec ready to accept such a PR? Maybe we should get more implementer commitments first?
• What's the name? I like .stream(), but if we do .arrayBuffer() returning a promise for an array buffer, maybe it's confusing for .stream() to return a non-promise? Maybe that argues for .toStream() or .asStream().

[annevk]

I think the main benefit of a protocol (similar to Symbol.iterator) is that a) userland objects can participate, b) we can potentially make it easier to add such functionality to future IDL classes, and c) we settle any future naming disputes.

[inexorabletash]

I can't make spec editor or implementer commitments at the moment so no
urgency on my part.

On Tuesday, September 27, 2016, Jimmy Karl Roland Wärting <
notifications@github.com> wrote:

I like .stream() without promise
It short!

You haven't thought of doing .toArrayBuffer() or .asArrayBuffer() so why
should .stream() be prefixed with to/as while arrayBuffer don't?

Its more like fetch. you get the stream from res.body and res.arrayBuffe()
as promise

So I don't think there's any confusion. I just think it's better to mimic
the Response method .arrayBuffer() .text()

If you want to get a stream synchronized you could still use the Response
hack I shown you initially (which would break the purpes of having
something like toStream() be prototyped to blob in the first place)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#40 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAvF2yAFadYnM-GM3VMeg0mRzxWQK8doks5quS8ZgaJpZM4Izqf9
.

[jimmywarting]

Now when i have tried converting mostly anything to a readable byte stream on a couple of things such as fetch, blob, formData, JSZip, TextEncoder/Decoder, Buffers, iterators, blinks sandboxed filesystem API, and even strings

I'm starting to thing that @annevk got a good point about symbol