Add hooks for observing navigation lifecycle from other specs

[jgraham]

Other specs want to be able to observe the document lifecycle as pages are loaded in navigation. In particular this is a requirement for implementing the navigate command in WebDriver-BiDi, and to provide the events automation clients require to monitor navigations initiated by the browser (e.g. when a link is clicked), It also seems like it will be a requirement for any attempt to standardise the Web Extensions API, and navigation-timing kind of monkey patches in these hooks by saying "if <condition> happens then do <action>". Whilst other specs could adopt that same model it seems like a better formalism for HTML to expose the navigation lifecycle as hooks that other specs can use.

For WebDriver-BiDi, which is the case I'm most familiar with, I think we'd want something like the following:

• Some kind of identifier for the navigation that can be used to associate particular events with a specific navigation. This could be done implicitly, but it might be easier for the HTML spec to define an identifying token for the navigation that can be passed down into other algorithms (in terms of Blink I assume this would map onto the loader id).
• A hook when the navigation starts i.e. just before the in-parallel section of navigate.
• A way to pass the navigation is into Fetch so that if/when we put similar lifecycle hooks there (also likely required for the same use cases) resource fetches that are associated with a navigation can be labelled as such.
• Some way for other resource loads initiated during parsing to figure out the current navigation (maybe it could just be an internal property of the browsing context?), so that they can also be traced back to the ongoing navigation.
• Hooks during the the final stages of parsing corresponding to the times when the readyState is changed.
• Hooks when the navigation fails.
• Hooks for non-fetch navigations or non-document response types indicating that the navigation is complete (I haven't thought carefully about the details here).

This isn't super-detailed, but at this stage I'm looking to see if people think this is the right approach.

[annevk]

ID is https://html.spec.whatwg.org/#concept-environment-id. This seems generally fine for observing, but if these hooks can also influence events then ordering starts mattering at which point they cannot be general-purpose hooks. (See how https://html.spec.whatwg.org/#update-the-rendering (a bit down) invokes a lot of endpoints in order to ensure ordering is defined.)

[jgraham]

Ah, I didn't think of the environment id. But it seems like that isn't created until https://html.spec.whatwg.org/#initialise-the-document-object which is well into the parallel steps of the algorithm and after various points of failure e.g. network errors.

I think the navigate integration points I'm imagining are all passive, and ordering probably doesn't matter, but we're eventually going to want to expose hooks in fetch that allow automation clients (and extensions?) to intercept the request and provide a custom response (or handle authentication challenges etc.). I can see that is going to require calling out to specific steps rather than a general event-broadcast like hook.

[annevk]

Oh right, it needs to be a separate ID. The environment ID is reset for redirects so that doesn't work.

[domenic]

I'm a little wary of adding general-purpose hooks given how poorly that's gone in the past (e.g. w3c/page-visibility#51). The alternative would be having HTML call out directly to WebDriver BiDi so that HTML knows all the specs that are depending on these integration points and they can get proper review. Did we consider and reject that route?

On the other hand, I don't see a technical problem with observing-only hooks. We probably would need strong language about only using them for observation, e.g. not allowing modification of the URL or status, or people doing non-rigorous things like "cancel the navigation with this ID" or similar. Maybe that'd be enough.

[jgraham]

I didn't really consider explicitly callign in to WebDriver here; I had the impression that integration points that look like events were prefered over ones that look like direct function calls. But if that's wrong we could certainly consider changing the style.

One reason to prefer the current style is that the integration points you need are likely to be very similar for WebDriver-BiDi, navigation-timing, WebExtensions (assuming that gets standardised), and perhaps others. Of course we can just call into all those specs directly but it's more work overall if we think that the requirements are similar enough to be handled with one set of changes to HTML. On the other hand at least WebDriver-BiDi and WebExtensions are going to want mutable access to requests/responses in fetch (for the request rewriting and interception APIs) so we're going to end up with direct calls to those specs anyway for that feature.

I agree it's a problem if specs monkeypatch things in a non-rigorous way. The way the patch is written the data passed into other applicable specifications is supposed to be immutable, so in theory that isn't a problem. I think adding a navigation id doesn't really exacerbate the problem of bad monkey patching much, because those specs which don't get the integration right could already say things like "cancel any ongoing navigation" without paying attention to the details (and indeed there's some argument that the navigate algorithm in HTML is already quite handwavy about this topic as it assumes that the presence of an ongoing navigation in a given browsing context is just something that's just ambiently known by the environment).

[jgraham]

@foolip are you able to look at this?

[foolip]

Until we have a concrete second spec wanting to integrate with these hooks, I think we can err on the side of narrower scope, and define hooks with names like "WebDriver BiDi navigation start steps". These could be hooks that BiDi can define somewhere in the spec, or maybe a direct call into BiDi. I lean in the favor of hooks because I suspect we're not going to get the name and arguments of those algorithms right from the start, and hooks allow for some small amount of monkey-patching of additional arguments internally to BiDi while things are in flux.

When a second concrete case comes along, we can turn it into either one generic hooks, an ordered list of hooks, or an ordered list of calls into the relevant specs.

[domenic]

The point about a single spec is well-taken. Given that I'd prefer these to be direct calls, instead of hooks.