Add a speculative flag to `Request` as well as related processing #881

yoavweiss · 2019-03-19T14:05:46Z

As discussed on w3c/resource-hints#74 and on whatwg/html#4115, this PR adds a speculative flag to Request, which makes sure the request:

Survives navigations
Is low priority
Can be ignored by the UA (haven't added that part yet)
Adds the related request headers

I intend (as part of whatwg/html#4115) to then make sure prefetch requests are sent with this flag set.

Preview | Diff

yoavweiss · 2019-03-19T14:06:53Z

Can be ignored by the UA (haven't added that part yet)

@annevk - any advice on how such language can be formulated?

annevk

I suppose you could return early somehow for a speculative fetch. What does ignoring mean though? Not invoking fetch at all?

fetch.bs

annevk · 2019-03-21T09:38:33Z

fetch.bs

@@ -5628,6 +5638,7 @@ interface Request {
  readonly attribute RequestRedirect redirect;
  readonly attribute DOMString integrity;
  readonly attribute boolean keepalive;
+  readonly attribute boolean speculative;


This would only result in setting the header? You'd still get the response and such?

Yeah, that's the current intention. Even though prefetch is destined for the next navigation, the current renderer can still get the response (e.g. onload fires for it).

Well, but this adds it to fetch() as a new feature. Was that intended? Or should this only be accessible via <link>?

I thought it'd be needed to make those headers survive SWs (although that may vary based on where #880 ends up)

But this would also allow creation. Surviving SW should happen based on copying private state I think.

^That sounds right to me. This is similar to the idea that priority is a concept that gets preserved here but is never exposed to the Fetch API.

fetch.bs

yoavweiss

Thanks for reviewing! :)

fetch.bs

yoavweiss · 2019-03-21T09:46:00Z

fetch.bs

@@ -5628,6 +5638,7 @@ interface Request {
  readonly attribute RequestRedirect redirect;
  readonly attribute DOMString integrity;
  readonly attribute boolean keepalive;
+  readonly attribute boolean speculative;


Yeah, that's the current intention. Even though prefetch is destined for the next navigation, the current renderer can still get the response (e.g. onload fires for it).

fetch.bs

yoavweiss · 2019-03-21T10:16:21Z

fetch.bs

+   <li><p>If a <var>request</var>'s <a for=request>speculative flag</a> is set, then the user agent
+   may <a for=fetch lt=terminated>terminate</a> the <a>fetch</a> and return an <a>aborted network
+   error</a>, in cases where it's reasonable to believe it will result in better user experience.
+


@annevk I wasn't sure what to write here, so started by writing something. It feels a bit fuzzy to be part of the processing model, so advice on making it better would be appreciated

How concrete is it in the implementation?

Currently I'm not aware of implementations that do that at the moment. When discussing whatwg/html#4115 @youennf wanted to give UAs the concrete possibility to do so, while staying conformant.

yoavweiss · 2019-03-21T10:23:38Z

I suppose you could return early somehow for a speculative fetch. What does ignoring mean though? Not invoking fetch at all?

I added a step to the processing model, aborting early at the UA's discretion. I think it'd be better to notify the developer that prefetch is being ignored.

youennf · 2019-03-22T23:05:30Z

Overall, I would tend to go with the minimum proposal to have a functional targeting navigation loads.
Some additional questions/observations:

Is prefetch sometimes/mainly/solely targeting (top) navigation loads.
It seems to be the case since we want prefetch (at least navigation cross origin prefetch) to escape cache partitioning for instance.
Preload, on the other hand, is about subresource loads, and can be used for all other cases.
I like the difference as this allows to express that:

a web page might know that it will probably navigate to another page (prefetch)
a web page might know its own resources (preloads)
a web page might not try to (in fact fail to) determine the resources of the potential next page and try to over-optimize the next page, especially third-party ones.

If we narrow down the scope of prefetch to navigation loads, we might be able to simplify things without loosing too much functionality.

Does it make sense for prefetch loads to go through SW of the prefetching page?
The response might depend on 1. Let's assume we target navigation loads.
For third-party navigations, SW interception benefit is unclear, downside is increased processing/latency.
For same-origin navigations, there might be some theoretical benefits.
That said, the service worker intercepting the prefetch might not be the one intercepting the navigation load. Preloads might be good enough there.

We also do not want to have the prefetch load go to the SW of the future navigation page.

Should a SW that intercepts the navigation load know that a prefetch is available?
We might expect the navigation load to go through SW before using the prefetch cache.
It could be handy for the SW to say: yes, go use the prefetch cache.
Might be dealt separately.
Do we need the boolean in the Request interface?
If we allow prefetch loads SW interception, probably. Otherwise, question 5.
Do we want JS to be able to create Request with this flag set?
If we narrow down the scope to navigation loads, I am not sure allowing fetch() with a speculative load is very useful. One use case might be to emulate form submissions.
Not sure the complexity of handling long living requests with body is worth this usecase.
Should responses be made available to the web page doing the prefetch?
onload/onerror might be fine to have for .
This does not require to provide the response to the webpage.
In general, we should be worried of communicating third party responses to the web page if mode is not cors. But cors might not be always suitable for prefetch. no-cors might not be suitable either, for instance due to CORP checks. Maybe prefetch would be better served with its own mode.

yoavweiss · 2019-04-15T14:58:55Z

@youennf thanks for your comments! :) (and apologies for my slowness, travel is my typical excuse)

Regarding 1-3, they are defined in whatwg/html#4115
Do you think we also need to add similar restrictions in Fetch?

Regarding 4 and 5, we can probably avoid those for the time being. The tradeoff here would be that prefetches would be somewhat magical, but maybe we can tackle that later?

Regarding 6, agree that there's no need to communicate the actual response bytes to the prefetching page. Not sure how to spell that out as a spec concept though.

youennf · 2019-05-17T16:37:13Z

To improve efficiency, a prefetch could also be allowed to consult the HTTP cache and/or prefetch cache.
That said, the onload/onerror event handlers may allow the web page triggering the prefetch to infer some user specific information. This might need some analysis.

jkarlin · 2019-05-17T17:24:46Z

Is prefetch sometimes/mainly/solely targeting (top) navigation loads.
It seems to be the case since we want prefetch (at least navigation cross origin prefetch) to escape cache partitioning for instance.

There are many cases where prefetch is used to load subresources. For instance, there are sites that will create a <link rel=prefetch> to critical-path subresources when you click on a link in order to save a RTT. I'd really like us to solve prefetch for both x-origin documents and subresources.

youennf · 2019-05-17T17:38:16Z

Subresources could potentially be optimized with link preloads in the prefetched resource or web packaging.

jkarlin · 2019-05-17T17:51:11Z

Link preloads in the document are still a full RTT slower than prefetching the subresource at navigation start.

youennf · 2019-05-17T18:29:14Z

Sure. In general though, it seems rather brittle for a page to try optimizing the loading of a cross-origin page that might change any minute. Also, if the subresource is already in the cache (say a JS library that is needed for all pages of the prefetched origin), that might actually slow things down to prefetch it if the subresource is already in the cache and prefetch is not allowed to look at the cache.
We can probably start with something simple that solve some of the main use cases and improve upon it.

jkarlin · 2019-05-17T18:54:31Z

Right. We'd need the prefetch to specify a cache key (destination origin) to use for the subresource, otherwise the prefetch could be wasted if the cache already has the resource

domfarolino · 2019-06-21T11:21:06Z

fetch.bs

@@ -1077,6 +1077,9 @@ explicitly set <a for=/>request</a>'s
 <p id=keep-alive-flag>A <a for=/>request</a> has an associated
 <dfn for=request export>keepalive flag</dfn>. Unless stated otherwise it is unset.

+<p id=speculative-flag>A <a for=/>request</a> has an associated
+<dfn for=request export>speculative flag</dfn>. Unless stated otherwise it is unset.


Probably this should be a boolean, as I think we're leaning more towards explicit booleans as opposed to "flags"s these days?

jkarlin · 2019-06-24T18:48:16Z

Sure. In general though, it seems rather brittle for a page to try optimizing the loading of a cross-origin page that might change any minute. Also, if the subresource is already in the cache (say a JS library that is needed for all pages of the prefetched origin), that might actually slow things down to prefetch it if the subresource is already in the cache and prefetch is not allowed to look at the cache.
We can probably start with something simple that solve some of the main use cases and improve upon it.

Sorry, I meant to reply to this earlier. This sounds good to me.

Add a speculative flag to Reqeust, and related processing

c05e3ec

yoavweiss requested review from annevk and youennf March 19, 2019 14:05

lint

e11eb15

yoavweiss changed the title ~~Add a speculative flag to Reqeust as well as related processing~~ Add a speculative flag to Request as well as related processing Mar 20, 2019

yoavweiss mentioned this pull request Mar 20, 2019

Specify processing model in terms of Fetch w3c/resource-hints#86

Closed

annevk reviewed Mar 21, 2019

View reviewed changes

yoavweiss commented Mar 21, 2019

View reviewed changes

Add somewhat fuzzy language about aborts

b693bd0

yoavweiss commented Mar 21, 2019

View reviewed changes

yoavweiss added 2 commits March 21, 2019 12:47

Try to make the bots happy

1debe2c

lint

477decf

yoavweiss mentioned this pull request Mar 21, 2019

Add prefetch processing model, including double-key caching privacy protections whatwg/html#4115

Closed

youennf mentioned this pull request May 17, 2019

Double-keyed HTTP cache #904

Closed

domfarolino reviewed Jun 21, 2019

View reviewed changes

domfarolino mentioned this pull request Jul 30, 2019

Prefetch request changes to improve privacy w3ctag/design-reviews#398

Closed

5 tasks

Base automatically changed from master to main January 15, 2021 07:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a speculative flag to `Request` as well as related processing #881

Add a speculative flag to `Request` as well as related processing #881

yoavweiss commented Mar 19, 2019 •

edited by pr-preview bot

yoavweiss commented Mar 19, 2019

annevk left a comment

annevk Mar 21, 2019

yoavweiss Mar 21, 2019

annevk Mar 21, 2019

yoavweiss Mar 21, 2019

annevk Mar 21, 2019

domfarolino Jun 21, 2019

yoavweiss left a comment

yoavweiss Mar 21, 2019

yoavweiss Mar 21, 2019

annevk Mar 21, 2019

yoavweiss Mar 22, 2019

yoavweiss commented Mar 21, 2019

youennf commented Mar 22, 2019

yoavweiss commented Apr 15, 2019 •

edited

youennf commented May 17, 2019

jkarlin commented May 17, 2019 •

edited

youennf commented May 17, 2019

jkarlin commented May 17, 2019 •

edited

youennf commented May 17, 2019

jkarlin commented May 17, 2019

domfarolino Jun 21, 2019

jkarlin commented Jun 24, 2019

Add a speculative flag to Request as well as related processing #881

Are you sure you want to change the base?

Add a speculative flag to Request as well as related processing #881

Conversation

yoavweiss commented Mar 19, 2019 • edited by pr-preview bot

yoavweiss commented Mar 19, 2019

annevk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yoavweiss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yoavweiss commented Mar 21, 2019

youennf commented Mar 22, 2019

yoavweiss commented Apr 15, 2019 • edited

youennf commented May 17, 2019

jkarlin commented May 17, 2019 • edited

youennf commented May 17, 2019

jkarlin commented May 17, 2019 • edited

youennf commented May 17, 2019

jkarlin commented May 17, 2019

Choose a reason for hiding this comment

jkarlin commented Jun 24, 2019

Add a speculative flag to `Request` as well as related processing #881

Add a speculative flag to `Request` as well as related processing #881

yoavweiss commented Mar 19, 2019 •

edited by pr-preview bot

yoavweiss commented Apr 15, 2019 •

edited

jkarlin commented May 17, 2019 •

edited

jkarlin commented May 17, 2019 •

edited