Encode and decode should take/produce streams

[marcoscaceres]

Talking to folks here at Mozilla, we would like it to be possible to use this API with the WHATWG streams API. Use case is to be able to upload while encoding, or decode while downloading via fetch etc.

+@domenic

[domenic]

So, I've looked in to this a decent bit. Basically, it is pretty reasonable and easy to do APIs of the form

(streaming input chunks) -> (big binary blob)

i.e. these two are achievable

(streaming encoded chunks) -> (decoded binary blob)
(streaming decoded chunks) -> (encoded binary blob)

More concretely, how this probably looks like is a writable stream with a result promise. Example, pre-bikeshedding, and applying some speculative fixes for open writable stream API issues:

const decoder = new ImageDecoder();
// assume it does sniffing on input
// alternately it could accept a target type as a constructor parameter and error if the input does not conform

decoder.result.then(img => {
  // here img is an ImageBitmap or ImageData or Blob...
  // bikeshed to determine which is most applicable
  // or, let the author choose via a constructor parameter
});

// Use case 1: manually assembling bytes and feeding them in
const writer = decoder.getWriter();
writer.write(chunk1); // chunks should be Uint8Arrays, or maybe any BufferSource
writer.write(chunk2);
...
writer.close();

/////////

// Use case 2: piping from a readable stream
// Here we use a fetch body stream as an example.

fetch(...).then(response => response.body.pipeTo(decoder));

Things are basically the same for the encoder example, where you feed it a stream of raw bytes and it gives you back a specified image type.

You probably also want conveniences like ImageDecoder.canDecode("image/png") or ImageEncoder.canEncode("image/jpeg"). Bikeshedding possibilities there: naming could be can(En|De)code[Type] and return the ""/"maybe"/"probably" values, or maybe for images things are a bit more straightforward than for video/audio, and we could just do .supports("image/png") returning a boolean.

---

The above is the most obviously doable API. There are other more complicated possibilities as well.

For example, some versions of image formats support streaming on both sides: you feed in some raw bytes to the encoder, and it progressively gives you parts of the resulting png. Or you feed in some encoded bytes to the decoder, and it gives you progressive scanlines of the jpeg. This would be a transform stream, with { writable, readable } sides. But it only applies to some variants of some image formats, and as the jpeg scanlines example shows, it can involve codec-specific information.

Another potentially-interesting high-level API is something like allowing imgEl.srcObject = readableStream, to bypass all this low-level stuff and just tell the browser "please get these bytes into my image in some way." This would be most useful for synthetic streams, instead of ones derived directly from fetch.

---

Hope this helps! I'm happy to start speccing this kind of thing if there's implementer interest.

[marcoscaceres]

@domenic we are really interested in this at Mozilla (and have someone who is willing to implement it). It would be awesome if you have time to help edit it.

[AshleyScirra]

This looks like a really good idea, but it seems it would involve mostly rewriting the spec to be based around ImageEncoder and ImageDecoder objects instead. I'm happy for that to happen. Do you want to draft that or shall I write something up? I'd add I'm not very familiar with streams though...

[domenic]

@marcoscaceres I can give this a shot, although it will be about a month before I can really get to it. I also need to circle back to those writable stream API changes which are currently just open bugs and not incorporated into the spec.

@AshleyScirra If implementers prefer, I think it'd be OK to continue with the current spec as-is, and later rephrase it on top of the lower-level streaming primitives. (Well, that's assuming you get around to writing the spec. Right now it lists a few methods but doesn't actually specify what they do.)

Alternately maybe implementers would prefer to wait for the streams stuff to shake out before doing any work on the higher-level sugar methods proposed here, in which case ... I'll try to hurry!

[marcoscaceres]

@domenic, I've done some work already setting up the specification to be algorithmic. See PR.

Sent from my iPhone

On Oct 15, 2015, at 3:51 PM, Domenic Denicola notifications@github.com wrote:

@marcoscaceres I can give this a shot, although it will be about a month before I can really get to it. I also need to circle back to those writable stream API changes which are currently just open bugs and not incorporated into the spec.

@AshleyScirra If implementers prefer, I think it'd be OK to continue with the current spec as-is, and later rephrase it on top of the lower-level streaming primitives. (Well, that's assuming you get around to writing the spec. Right now it lists a few methods but doesn't actually specify what they do.)

Alternately maybe implementers would prefer to wait for the streams stuff to shake out before doing any work on the higher-level sugar methods proposed here, in which case ... I'll try to hurry!

—
Reply to this email directly or view it on GitHub.