Specify how media elements make and validate range requests

[jakearchibald]

I'm looking for early feedback on this. I'm going to add some comments to highlight parts I'm not sure about.

---

💥 Error: Wattsi server error 💥

PR Preview failed to build. (Last tried on Jan 21, 2021, 12:09 PM UTC).

More

PR Preview relies on a number of web services to run. There seems to be an issue with the following one:

🚨 Wattsi Server - Wattsi Server is the web service used to build the WHATWG HTML spec.

🔗 Related URL

Parsing MDN data...
Parsing...
Generating HTML variant...
Error: missing <dfn> for topic "concept-request-type" explicitly from <span> element containing "type"; previous heading contents are "4.8.12.5 Loading the media resource"
Error count: 1
Saving index-html
Splitting...
Saving index.html
Saving introduction.html
Saving infrastructure.html
Saving common-microsyntaxes.html
Saving urls-and-fetching.html
Saving common-dom-interfaces.html
Saving structured-data.html
Saving dom.html
Saving semantics.html
Saving sections.html
Saving grouping-content.html
Saving text-level-semantics.html
Saving links.html
Saving edits.html
Saving embedded-content.html
Saving images.html
Saving iframe-embed-object.html
Saving media.html
Saving image-maps.html
Saving embedded-content-other.html
Saving tables.html
Saving forms.html
Saving input.html
Saving form-elements.html
Saving form-control-infrastructure.html
Saving interactive-elements.html
Saving scripting.html
Saving canvas.html
Saving custom-elements.html
Saving semantics-other.html
Saving microdata.html
Saving interaction.html
Saving dnd.html
Saving browsers.html
Saving window-object.html
Saving origin.html
Saving history.html
Saving browsing-the-web.html
Saving webappapis.html
Saving dynamic-markup-insertion.html
Saving timers-and-user-prompts.html
Saving system-state.html
Saving imagebitmap-and-animations.html
Saving comms.html
Saving server-sent-events.html
Saving web-sockets.html
Saving web-messaging.html
Saving workers.html
Saving worklets.html
Saving webstorage.html
Saving syntax.html
Saving parsing.html
Saving named-characters.html
Saving xhtml.html
Saving rendering.html
Saving obsolete.html
Saving iana.html
Saving indices.html
Saving references.html
Saving acknowledgements.html
Generating DEV variant...
Splitting...
Saving index.html
Saving introduction.html
Saving infrastructure.html
Saving common-microsyntaxes.html
Saving urls-and-fetching.html
Saving common-dom-interfaces.html
Saving dom.html
Saving semantics.html
Saving sections.html
Saving grouping-content.html
Saving text-level-semantics.html
Saving links.html
Saving edits.html
Saving embedded-content.html
Saving images.html
Saving iframe-embed-object.html
Saving media.html
Saving image-maps.html
Saving embedded-content-other.html
Saving tables.html
Saving forms.html
Saving input.html
Saving form-elements.html
Saving form-control-infrastructure.html
Saving interactive-elements.html
Saving scripting.html
Saving canvas.html
Saving custom-elements.html
Saving semantics-other.html
Saving microdata.html
Saving interaction.html
Saving dnd.html
Saving browsers.html
Saving window-object.html
Saving origin.html
Saving history.html
Saving browsing-the-web.html
Saving webappapis.html
Saving dynamic-markup-insertion.html
Saving timers-and-user-prompts.html
Saving system-state.html
Saving imagebitmap-and-animations.html
Saving comms.html
Saving server-sent-events.html
Saving web-sockets.html
Saving web-messaging.html
Saving workers.html
Saving worklets.html
Saving webstorage.html
Saving syntax.html
Saving named-characters.html
Saving xhtml.html
Saving obsolete.html
Saving indices.html
Saving references.html
Saving acknowledgements.html
Generating SNAP variant...
Error: missing <dfn> for topic "concept-request-type" explicitly from <span> element containing "type"; previous heading contents are "4.8.12.5 Loading the media resource"
Error count: 1
Saving index-snap

If you don't have enough information above to solve the error by yourself (or to understand to which web service the error is related to, if any), please file an issue.

[jakearchibald]

See #2813 - we may need to change the name of this.

[annevk]

I like "uses opaque response flag". We don't go more granular thus far than a Response for that kind of data and I doubt we'll ever will so it seems reasonable?

[jakearchibald]

Should this be an ordered set? It doesn't really matter in terms of behaviour, but an implementation using a set would use less memory.

[annevk]

Say you have: URL -> URL2 -> URL where the first URL redirects but the last one doesn't due the a cookie. Would it matter if that ends up as (URL, URL2)?

[jakearchibald]

I don't think so. The order isn't important here either.

I want to ensure that all chunks of a particular media resource were requested from the same endpoint if opaque responses are involved. If the server redirects, that's fine. If all the responses are not-opaque, anything goes.

The attack I'm wanting to avoid is:

1. Video src is https://personal.example.com, which contains personal data and is not a valid media element.
2. Media element issues request for 0-.
3. Service worker responds with 0-100 from a resource which is a known to be a video header, where the 101st byte represents the media length.
4. Media element issues request for 101-.
5. Service worker responds with fetch(event.request).

Since we've already given the media element a valid video header, the additional bytes from https://personal.example.com may leak as the video length etc etc.

However, this PR would see that the initial response url of null doesn't match https://personal.example.com, and return a network error.

[annevk]

So why would you store a list of URLs? Wouldn't you just stay opaque the moment you encounter it once?

[jakearchibald]

"opaque" isn't enough information. All the responses could be opaque but from different places. As a result, a load of non-video data could become valid video data and leak information.

I guess I could replace this with "initial response url list", which is the url list of the first response used, and a "mixed response sources flag".

Then, if the first entry of an additional response's url list doesn't match the first entry in the "initial response url list", set the "mixed response sources flag".

If the "uses opaque data flag" is set, or the response is opaque, and "mixed response sources flag" is set, return a network error.

[annevk]

I see.

[jakearchibald]

I think I'm breaking HLS by doing this. Will rethink my approach. I think "If opaque data is used, every response's (including ones previously used by this resource) first entry in the url list must match its request url" works.

[jakearchibald]

Should be fine now

[jakearchibald]

Those last few lines are adapted from some text that already existed. I'm a little concerned about how hand-wavey they are, especially as the canvas spec seems to contradict it (#2813).

[domenic]

The last few lines are allowed to be hand-wavey since they are statements of fact, not normative requirements. However they are not allowed to be wrong, so #2813 may indeed be a problem. I think it's OK for now though, pending getting #2813 figured out?

[jakearchibald]

Yeah, the text here matches what browsers do. The canvas section is wrong.

[jakearchibald]

Again, I'm worried about the hand-waving here, but I'm not sure how to improve it without defining how each codec works.

[annevk]

How does fetching behavior relate to the codec?

[jakearchibald]

1. Video starts fetching.
2. Container/codec decoding happens.
3. Through knowledge of the codec, the browser knows the metadata sits in the last 1000 bytes of the resource.
4. Fetch Range: 123456-

Or

1. User seeks resource to 10 minutes in.
2. Through knowledge of the codec, the browser knows the content at 10:00 sits 9000000 bytes in.
3. Fetch Range: 9000000-.

Without defining "Through knowledge of the codec", I'm just providing the steps for making a range request, but never really defining when it's called.

[annevk]

It might be good to call out that codec contract a little bit. So it's more clear what we're not defining. That will also make it easier for folks to tell us we're making the wrong assumptions.

[jakearchibald]

👍

[jakearchibald]

I've written this like an inner function that can access variables defined in the parent steps. Is this in any way ok?

[annevk]

Probably a question for the Infra Standard, but I think generally we try to avoid that and pass parameters explicitly.

[jakearchibald]

I placed it within the resource fetch algorithm because that runs for the entire load of the media element. As in, it only reaches the final step if the whole media finishes loading.

[annevk]

@domenic what do you think? I think personally I'd rater avoid nested functions.

[domenic]

Given the way in this this is referenced, it seems totally fine to me. It's barely a function; it's more annotating some steps by saying /* begin ranged fetch steps */ ... /* end range fetch steps */.

[jakearchibald]

I think I'm going to move these "creating a range header" steps into the fetch spec, to avoid these lines being duplicated in every spec that wants to make a range request.

[annevk]

If there's at least two callers (or expected to be) that seems reasonable.

[jakearchibald]

It'll be needed for downloads, and probably background fetch.

[jakearchibald]

I wasn't really sure how to build this header. Is "append" correct here?

[annevk]

For building a byte sequence that seems fine. There's some debate still over whether they should maybe be immutable, but this will probably end up working.

[jakearchibald]

Wait, this is wrong. brb

[annevk]

Do you remember why you said this was wrong? I am seeing the same in Chrome. Firefox is more lenient (though does require 2xx). Not sure exactly what Safari wants here, I cannot get 200 to work there.

[jakearchibald]

I can't remember why this is wrong. Safari doesn't like 200 here, but I think it should work (as described in the steps below).

There are other issues with this PR though, which I'll detail in a follow-up comment.

[jakearchibald]

Can I refer to first-byte-pos like this? It's from https://tools.ietf.org/html/rfc7233#section-4.2.

[annevk]

It might be good to cross-reference that term. Given the tight integration with HTTP uplifting to Fetch makes more sense to me now, even if this is the only caller.

[jakearchibald]

Is this too much detail for a note? It's difficult since the process of aborting fetches isn't really covered.

[domenic]

I think it needs to not use should and may in a note. You can e.g. substitute "encouraged" and "could". Otherwise it does seem helpful.

[jakearchibald]

@foolip Could you take a look at this?

[foolip]

Going on vacation soon and won't have time to finish all things, like this. I'll check back early in August to see if this is still open.

[domenic]

Nit: s/method/algorithm, I think.

[domenic]

Nit: <dfn goes on previous line. The Great Re-Wrapper can help by turning unwrapped lines into wrapped ones.

[jakearchibald]

I wasn't aware of that tool, cheers!

(fwiw I'd rather we just relied on editors for this kind of wrapping. Breaking mid-line makes it hard to search for terms across the document)

[domenic]

Do you think media resource is the right place to store these things? So far in the spec it seems less like a spec data structure with fields and more like a concept. Maybe putting them on the media element might be more appropriate.

[jakearchibald]

I use media resource as it changes when the src changes. If that happens, we want the properties to reset.

[domenic]

Fair enough.

[domenic]

Since we're trying to move away from flags and to booleans, maybe this should be a boolean? OK either way though; maybe we should only do an en-masse conversion instead of mixing up both into the spec.

[jakearchibald]

I wasn't aware of the flag->boolean change. Happy to change it here.

[domenic]

What is the relation between this request and the one possibly created by the range request steps? It seems like we're normatively requiring this one, and then if the browser wants to do a range request, it needs to do a second request. Is that correct?

[jakearchibald]

Yeah, that's what I intended. It feels like browsers should make a non-range request first to get the start of the resource, but also to determine if ranges are accepted.

[domenic]

It is a bit surprising we append the first, not the last (i.e. not the response's url). A note explaining that we are intentionally doing this for the purpose of tracking redirects would be good, IMO.

[domenic]

The last few lines are allowed to be hand-wavey since they are statements of fact, not normative requirements. However they are not allowed to be wrong, so #2813 may indeed be a problem. I think it's OK for now though, pending getting #2813 figured out?

[domenic]

I have a separate worry about the hand-waving here, which is that it gives the user agent license to do literally anything. For example a user agent could issue range requests but not use the ranged fetch steps. Maybe the intent is clear enough though that we don't need to change it.

[jakearchibald]

I'm worried about the "streaming protocol" bit, in that it might not follow the rules in terms of mixing opaque & non-opaque responses.

[domenic]

This would be clearer as two steps IMO.

[domenic]

I think it needs to not use should and may in a note. You can e.g. substitute "encouraged" and "could". Otherwise it does seem helpful.

[jakearchibald]

I'm going to move on to range requests & downloads, but in the mean time… any ideas how I could write tests?

I think all browsers will issue ranges requests for a simple wav file that's seeked, which I can automate in browsers that support autoplay. However, the spec doesn't require browsers to make range requests, so it feels like a bad approach. Is there any kind of "unable to test" warning I could create in this situation.

Or, should I make it a manual test, where you have to seek a media element to generate range requests?

[jakearchibald]

@annevk Is there a better way to link to this? The HTTP spec doesn't allow linking to the term directly.

[domenic]

Well, it should be abstracted out into a <span> with the appropriate <dfn data-x-href=""> in the dependencies section, for one, but that's just editorial...

[annevk]

Yeah, other than that I was thinking that maybe this should be in Fetch given the tight integration with HTTP?

[jakearchibald]

Wouldn't that mean duplicating HTTP's definition?

[annevk]

We'd still reference HTTP directly in Fetch as you did here, but for everyone else in the platform we'd have it abstracted.

[jakearchibald]

It's be nice to have a header list algorithm that took a header name, and returned the parsed parts as defined by HTTP. It might be nice to expose this on the headers object too.

Eg: Let parsedRange be the result of parsing a header given headerList and Range.

Where parsedRange would be a structure like:

[
  {
    "ranges-specifier": "bytes=0-",
    "byte-ranges-specifier": "bytes=0-",
    "bytes-unit": "bytes",
    "byte-range-set": "0-",
    "suffix-byte-range-spec": null,
    "byte-range-spec": "0-",
    "first-byte-pos": "0",
    "last-byte-pos": null
  }
]

[annevk]

That would be great, but that would only work for headers the browser supports (and might be tricky given where that code lives).

[domenic]

I think all browsers will issue ranges requests for a simple wav file that's seeked, which I can automate in browsers that support autoplay. However, the spec doesn't require browsers to make range requests, so it feels like a bad approach. Is there any kind of "unable to test" warning I could create in this situation.

In general it's OK to write tests that aren't fully required by the spec. This is especially true if all browsers pass them. (Or intend to pass them, e.g. all use range requests.)

[jakearchibald]

No longer a WIP as such, but I need to fix the first-byte-pos reference.

Will pick this up again soon.

[jakearchibald]

There are security issues with this PR, especially the way it checks only the first response URL.

The final response URL is the important one, although there might need to be further restrictions if the redirect goes cross-origin then comes back to the same final url, unless we want that to be observable.

whatwg/fetch#144 (comment) is a better summary of the issues and solutions.

Background fetch has some algorithms for validating partial responses, which might be useful here, although those are CORS requests https://wicg.github.io/background-fetch/#validate-partial-response-algorithm, the validation for opaque responses will need to be stricter.

I'm happy for this PR to close. It's probably easier to start again.