Navigation and session history rewrite

[jakearchibald]

This monster completely rewrites everything to do with navigation and traversal.

It introduces the "navigable" and "traversable navigable" concepts, which take on many of the roles that browsing contexts previously did, but better. A navigable can present a sequence of browsing contexts, which to the user seem to all be the same, but due to browsing context group switches, have different WindowProxys and are allocated in different agent clusters. A traversable navigable manages the session history for itself and all its descendant navigables, providing a synchronization point and source of truth.

The general flow of navigation and traversal is now geared toward creating a session history entry, populated with the appropriate document, before finally applying the history "step". The step concept for session history, managed by the traversable, replaces the previous idea of joint session history, which was a sort of deduplicated union of individual session histories for each browsing context within a top-level browsing context.

Notable things to still do before merging:

• Replace more browsing context references with navigables, with special care and attention to tree traversal and relationships
• Write nice introductions and overviews of the flow
• Rearrange everything into logical sections.
• Insert lots of "Jake diagrams" to illustrate tricky cases.
• Finish up remaining minor TODOs left in the text.
• Compare old IDs with new ones and preserve them where it makes sense. (Audit)
• Address outstanding issues in Navigation & session history rewrite notes
• Write or update tests, especially for cases called out in the above issues where we now have a clear answer. (We don't plan to be completely exhaustive yet; adding tests will be an ongoing effort.) (Audit)

Notable things we won't tackle this round, but are much easier to tackle in the future:

• Iframe restoration on (non-bfcache) history traversal.
• Overlapping navigations and traversals (see What browsers do for overlapping navigations/history traversals #6927) are probably not perfect yet (although this PR makes them better).
• Browsing context names (see Disowning owner should perhaps affect link targeting #313) are not perfect yet, although this PR makes them better.

Closes #854 by clarifying the javascript: URL origin and origin-checking setup.

Closes #1073 by properly resetting active-ness of documents when they are removed.

Closes #1130 by removing the source browsing context concept, using a sourceDocument argument instead, and taking source snapshot params at the appropriate early time.

Closes #1191 by properly sharing document state across documents, as well as overlapping same-document navigations plus cross-document traversals.

Closes #1336 by properly handling child browsing contexts.

Closes #1382 by only unloading after we are sure we have a new document (i.e., not a 204 or download).

Closes #1454 by rewriting session history closer to what implementations do, with the nested history concept in particular taking care of the issues discussed there.

Closes #1524 by introducing the POST data concept and storing it in the document state.

Closes #2436 by rewriting the spec for history.go() to be clear about the results. Tests: web-platform-tests/wpt#36366.

Closes #2566 by introducing an explicit "history object" definition. Tests: web-platform-tests/wpt#36367.

Closes #2649 through clear creation of srcdoc documents, including during history traversal.

Closes #3215 by preserving POST data and reusing it on reloads.

Closes #3447 by specifying a precise mechanism (the ongoing navigation) for canceling navigations, and the points at which that mechanism is consulted. It also stops queuing a task for hyperlink navigations.

Closes #3497 by posting appropriate tasks for cross-event-loop navigations.

Closes #3615 by rewriting traverse a history by a delta, which eventually calls into apply the history step, to navigate all relevant navigables.

Closes #3625 by storing information in the document state (not just the URL), so that future traversals can reconstruct the request appropriately.

Closes #3730 by doing proper task queuing for navigation, including one for javascript: URLs but not including one for normal same-frame navigations. Tests: web-platform-tests/wpt#36358.

Closes #3734 by rewriting the definition of script-closable to use well-defined concepts.

Closes #3812 by removing all uses of "active document" as a predicate instead of a property.

Closes #4054 by introducing the session history traversal queue and renaming the previous "history traversal task source" to "navigation and traversal task source".

Closes #4121 by doing the "allowed to navigate" check at the top of apply the history step.

Closes #4428 by keeping a strong reference from documents (including bfcached documents) to their containing browsing context.

Closes #4782 by introducing the top-level traversable and navigable concepts.

Closes #4838 by doing sandbox checking in a much more precise manner, in particular snapshotting the relevant flags early in any traversals.

Closes #4852 by using document state (in particular history policy container, request referrer, and request referrer policy) in reloads.

Closes #5103 by properly restoring scroll positions for everything that is traversed, as part of properly traversing more than one navigable.

Closes #5350 by properly restoring window names across browsing context group switches, and going back to the same browsing context as was previously there when traversing back across a BCG switch boundary. (Implementations could create new browsing contexts, as long as they restore the WindowProxy scripting relationships and other browsing context features; the result is observably equivalent.)

Closes #5597 by rewriting "allowed to download" to just take booleans, derived from the appropriate snapshotted or computed sandboxing flags.

Closes #5767, modulo bugs and oversights we made, by rewriting everything :).

Closes #5877 by respecifying "fully active" in terms of navigables, instead of browsing contexts.

Closes #6446 by properly firing beforeunload to all descendant navigables, although whether or not they actually prompt still allows implementation leeway.

Closes #6483 by introducing the distinction between current session history entry and active session history entry.

Closes #6514 by settling on using a single origin for these checks.

Closes #6628 by storing window.name values in the document state, so even in strange splitting situations like described there, they remain.

Closes #6652 by no longer changing history.state when reactivating a document from bfcache ("restore the history object state" is called only when documentsEntryChanged is true). Tests: web-platform-tests/wpt#36368.

Closes #6773 by having careful handling of synchronous navigations during traversals. Test updates: web-platform-tests/wpt#36364.

Closes #6798 by treating javascript: URL navigations as replacements.

Works towards #6809 by storing srcdoc resources in the document state.

Closes #6813 by storing referrer in the document state. Tests for the repopulation case: web-platform-tests/wpt#36352. (No tests yet for the reload case.)

Closes #6947 by rolling its contents into this change: PDF documents are put in the same category as other inaccessible, no-DOM documents.

Closes #7107 by clearing history state on redirects and when origin changes by other means, such as CSP.

Closes #7441 by making window.blur() a no-op because that was simpler than updating it to operate on navigables.

Closes #7722 by incorporating its contents into the rewritten version.

Helps with #8395 by at least ensuring the javascript: case does not fire beforeunload. Tests: web-platform-tests/wpt#36488. (The other cases remain open for investigation and testing.)

---

/browsers.html ( diff )
/browsing-the-web.html ( diff )
/canvas.html ( diff )
/common-microsyntaxes.html ( diff )
/comms.html ( diff )
/custom-elements.html ( diff )
/dnd.html ( diff )
/dom.html ( diff )
/dynamic-markup-insertion.html ( diff )
/embedded-content-other.html ( diff )
/embedded-content.html ( diff )
/form-control-infrastructure.html ( diff )
/form-elements.html ( diff )
/iana.html ( diff )
/iframe-embed-object.html ( diff )
/imagebitmap-and-animations.html ( diff )
/images.html ( diff )
/index.html ( diff )
/indices.html ( diff )
/infrastructure.html ( diff )
/input.html ( diff )
/interaction.html ( diff )
/interactive-elements.html ( diff )
/introduction.html ( diff )
/links.html ( diff )
/media.html ( diff )
/microdata.html ( diff )
/obsolete.html ( diff )
/parsing.html ( diff )
/references.html ( diff )
/rendering.html ( diff )
/scripting.html ( diff )
/sections.html ( diff )
/semantics-other.html ( diff )
/semantics.html ( diff )
/server-sent-events.html ( diff )
/structured-data.html ( diff )
/syntax.html ( diff )
/system-state.html ( diff )
/timers-and-user-prompts.html ( diff )
/urls-and-fetching.html ( diff )
/web-messaging.html ( diff )
/webappapis.html ( diff )
/webstorage.html ( diff )
/workers.html ( diff )
/worklets.html ( diff )
/document-lifecycle.html ( diff )
/document-sequences.html ( diff )
/nav-history-apis.html ( diff )

[domenic]

This is looking pretty reasonable; no real concerns!

At some point it'd be good to have a series of examples of how these concepts relate and manifest in the real world, perhaps first in GitHub comments, then eventually in the spec with diagrams and pictures. For example, I'd be interested in seeing the browsing session/navigable/session history diagram for scenarios like:

• Two independent top-level tabs, each of which navigates around
• One top-level tab with two iframes, each of which navigates around, and then the top-level tab also navigates
• Any interesting scenarios that illustrate tricky cases where the current set of concepts (browsing context/joint session history) are not sufficient

[jakearchibald]

I'd be interested in seeing the browsing session/navigable/session history diagram

Yeah, I want to create some table-like diagrams, like in the "Specifying the history as a 'timeline'" section of "#5767.

[jakearchibald]

Since we seem to be arriving at some kind of consensus in #6356, I'm going to refactor some stuff. @domenic, shout up if any of this sounds bad:

Currently a "browsing session" has a navigable, and some additional state. Instead, I'm going to make it a subclass of navigable, a "top level navigable". It will have the same state as browsing session currently has, except a navigable, since it'd be that navigable.

A "browsing session" will have associated session storage, as it's currently defined. A browsing context group will have a browsing session that cannot change throughout the life of the group. Multiple browsing context groups may share the same session.

Manual navigations to cross-origin URLs will always create a new browsing context group with a new session. Other navigations may create a new browsing context group (depending on cross-origin isolation) but they will have the same session.

[jakearchibald]

I got into a bit of a mess with this in terms of which threads could read & write from which data.

I now think that the top-level navigable needs to have its own copy of session history as a list of trees, so it can figure out things like history length without having to post tasks with every navigable to read that information.

This session history copy will be serialisable, which we need anyway for restoring session history to iframes on reload.

The copy will become temporarily out of date when session history is modified in a synchronous way, but a parallel task will be queued to keep it up to date. Here are the synchronous things:

• Adding a nested navigable (doesn't change history length or step, but just needs added to the tree)
• Removing a nested navigable (may change history length, but not step)
• Hash change / pushState navigations (changes history length & step)

I just need to make sure that, if a navigable makes 5 synchronous changes, it ignores all the synchronisations from the top level until it gets to the one that concerns its latest synchronous change.

@domenic these two copies with strict rules around which thread can access which should make it easier for your new history API to have synchronous access to its own session history.

[domenic]

I now think that the top-level navigable needs to have its own copy of session history as a list of trees, so it can figure out things like history length without having to post tasks with every navigable to read that information.

Interesting. My understanding of how this is implemented, generally, is that the "browser process" has the list of trees, and individual frames' processes only have the length value. All navigations go through the central browser process which pushes updates (presumably with some races) to length, state, etc.

/cc @natechapin for his thoughts, as he's been looking at this recently and can correct me if I've got this wrong.

I just need to make sure that, if a navigable makes 5 synchronous changes, it ignores all the synchronisations from the top level until it gets to the one that concerns its latest synchronous change.

This seems reminiscent of the interop issues in this document.

@domenic these two copies with strict rules around which thread can access which should make it easier for your new history API to have synchronous access to its own session history.

I agree in theory, although I worry about the potential mismatch with implementation in practice...

[jakearchibald]

I agree in theory, although I worry about the potential mismatch with implementation in practice...

Ok, I won't over-index on those requirements. We can figure out what should be copied across & when some other time.

[jakearchibald]

I've been playing around with some pseudocode to get a feel for the shape of the algorithms. Right now it's roughly like this:

New navigations select a target context then go to navigate.

History traversals go to traverse the history by a delta.

Navigate handles:

1. Perform synchronous navigations via navigate to a fragment, then traverses the history to the new entry. Otherwise:
2. Prompt to unload, if necessary.
3. Fetch the target url (if necessary) and process the response. Failures here either no-op (203) or create an error page.
4. Unload the current document, create the new history entry.
5. Traverse the history to the new entry.

Traverse the history to an entry handles:

1. If the entry's document is null, then navigate it. Otherwise:
2. Introduce the new page (visibility events and such)

Traverse the history by a delta handles:

1. If the target history entry has a document, and it's different to the current document, prompt to unload (if necessary), and unload.
2. Traverse the history to the target entry.

Issues with this approach:

• The way navigate calls traverse which can call navigate is weird and circular. It never loops in practice, but it makes for confusing reading.
• Navigate handles all "create a new document for a history entry" cases, but in reality we need to handle the failure cases for "link click" and "back button" differently.
• "Traverse the history by a delta" assumes that a history traversal will only impact one navigable.
• Unloading happens in multiple places, and preventing it happening twice for a single document is fragile (and kinda wrong in the current spec).

---

Here's what I'm going to try and do:

"Navigate":

1. Check if the navigable can be navigated by the initiator
2. If the navigation should be synchronous, then:
   1. Create new entry, change it synchronously
   2. Update history length/index synchronously in the current agent
   3. Queue a session history task to:
      1. Add/replace the new entry to the session history tree.
      2. "sync to a history step" (this updates history length elsewhere and resolves conflicts)
3. Otherwise:
4. Check for an unload prompt. If one is requested, show it. If the user cancels, then return, otherwise mark the document as "cleared for navigation".
5. Create new history entry.
6. "Attempt populate history entry". If it doesn't create a document or an error page, abandon the navigation.
7. Queue a session history task to:
   1. Add/replace the new entry to the session history tree.
   2. "sync to a history step" (this updates history length elsewhere and resolves conflicts)

"Attempt populate history entry":

1. Fetch the target url (if necessary) and process the response.
2. If the result is Content-Disposition, start the download and return.
3. If the result is a 203/4, return.
4. If the result is a network error, populate the history entry with an error and return.
5. Populate the history entry with a document and return.

"Sync to a history step":

1. Look at tree to determine history length and index.
2. Queue tasks with active documents:
   1. Update local history length and index.
   2. If this navigable does not need to change history entry, abort these steps.
   3. If the traversal will result in changing documents:
      1. If the document isn't "cleared for navigation", check if the navigable can be navigated by the initiator, and check for unload prompt and show it if necessary. If the user cancels the prompt, then do whatever we decide in Nested contexts and unload prompts #6446.
      2. If doc in new entry is null, then "attempt populate history entry" for the entry. If it doesn't create a document or an error page, do whatever we decide in What should happen when traversing to a history entry results in nothing displayable #6483. Otherwise:
      3. Unload current doc.
      4. Introduce new doc.

Any red flags there?

[EmmanuelFm]

(spam)

[domenic]

Generally no red flags. I must admit I'm a bit sad that just after I feel like I've got the navigation algorithm straight in my head, you plan to change it dramatically, but your "issues with this approach" (2)-(4) are compelling.

Check if the navigable can be navigated by the initiator

You might get yourself into trouble here with "initiator": that's #1130. Maybe you can just preserve the existing brokenness there, or maybe I can try to fix some of it to prepare the way.

[jakearchibald]

Yeah, I saw the hand-waving with "initiator" 😞. I was going to leave it vague in this first pass. Or maybe it'll bother me too much. Thanks for the link, I didn't realise there was an existing issue for this!

[jakearchibald]

@domenic I've just done my read-through & tweaking of "clicking on a hash-navigation link" if you want to review that path too.

I'll let you know as other paths are ready for review.

[domenic]

I traced through that path as best I could, and probably commented on some stuff that isn't applicable for such cases as well. I think the trickiest things I found were:

• Be careful with document's URL vs. session history entry URL
• I don't understand "nested histories"

Otherwise it's mostly minor requests for clarification or suggested touchups.

[jakearchibald]

I'm currently looking at rearranging the path of "navigate" a bit.

Right now the spec follows this pattern:

Navigate unloads the current document, then, calls out to process a navigate fetch, process a navigate response or process a navigate URL scheme.

Process a navigate fetch handles redirects and such, and calls process a navigate response.

Process a navigate response handles CSP, 204, 205, download triggering, then calling out to the correct handler for the type.

Process a navigate URL scheme is a little hand-wavey and broken.

The handler for the type (eg plain text file) takes care of creating the document, then calling update the session history.

Creating the document selects the browsing context (following COOP), records navigation timing, and handles declarative refreshing.

Update the session history does what you'd expect.

It feels odd that this is pretty much a sequence of calls. Because of this it requires a bag of state to be passed around. You also get weirdness like the document creation steps handling navigation timing.

I'm going to try and make it like this:

Navigate:

1. Unload the current documents.
2. Create a history entry and pass it to populate history item if input is a fetch URL, otherwise populate history item with response if input is a response.
3. If the history entry's document is null (indicating failure to create something renderable, eg 204), return.
4. Wait for document to start parsing, but before scripts.
5. Queue into the traversables session history queue, update the history tree, and apply the history step.

Populate history item fetches the entry's url, handles CSP, 205, 205, and downloads, otherwise calls populate history item with response with the history entry and response.

Populate history item with response creates the document depending on the response type, and records navigation timing, switches the browsing context in the history entry if needed, and handles declarative refreshing.

In this case it feels like "navigate" handles the navigation, and calls out to other things for part of that, rather than it being a sequence of calls that each hand off to another.

I'm hoping this will reduce the amount of state that needs to be passed around, making things easier to follow. It might even turn out that the history entry itself is enough state.

[domenic]

Hmm. I have a bit of status quo bias because I've spent a good amount of time fixing up and understanding the existing algorithms, but I don't quite see why your proposed alternative is better.

In particular, what I find intuitive about the current spec is that, at a high level, there's a pipeline: input -> response -> document -> update history. Each step feels nicely separated and feeds into the next, as e.g. we can see by how "process a navigate response" gets reused.

Your new proposal seems confusing to me in that it starts creating history entries before we even have a response, much less a document. It seems like it's overloading the concept of history entry to be two things: a bag of state that may one day turn into a real history entry, and then eventually a real history entry. In other words, I like how in the current spec history entries are only ever created right before they're put into a real session history list.

Also just on a naming level, I think "populate history item" is much less clear than "process a navigate fetch".