History and navigation, including the Navigation API

[domenic]

Description

I'd like to propose a focus area for Interop 2023 focused on session history and navigation between URLs. This would include:

• Foundational interoperability work on things like: what happens when navigating (especially in edge cases); how iframes impact the session history; when load/hashchange/popstate/etc. events fire; the sync vs. async nature of history traversals; what happens if code tries to go back and then forward and then navigate; how well about:/data:/blob: URLs work for navigation; scroll restoration; fragment navigation; etc.

• The existing navigation and history APIs, i.e. window.location and window.history.

• The new navigation API, which provides a much cleaner model for history and navigation. (This spec is currently in the WICG, but will be merged into the HTML Standard in October.)

Rationale

These areas are historically a large pain for web developers. @tbondwilkinson wrote up a vivid example; you can also see the HTML Standard's list of issues tagged "interop" and either "topic: navigation" or "topic: session history". This pain often spills over into broken behavior for users, e.g. pages not loading, the back button not working, scrolling being broken, or even jQuery's file upload code failing due to interaction with javascript: URL iframes. I suspect implementers who have worked in this area, and been on the frontlines of relevant bug triage, know how thorny an area this is (cc @annevk, @hsivonen, @smaug----, @natechapin). Getting interop on the foundations and existing APIs, and encouraging implementation of the new navigation API, will solve a lot of developer pain.

For some aggregate signals, for the navigation API we have many upvotes on the original WICG proposal (the second-most of any WICG proposal to date). For general history and navigation, we have use counters such as ~30% of pages using history.replaceState(), ~20% of pages using sandboxed iframes, or of course the fact that 100% of page loads happen via navigation :).

Now is a good time to work on this project. On the specification side, within the month of October we plan to land whatwg/html#6315, which will resolve ~45 known spec issues with a definitive, principled behavior based on a coherent model. As part of that, and in the month or so after landing, we'll be increasing the test coverage for navigation and history, ensuring that all existing tests match the updated spec. Additionally, once that PR is landed, we're planning to move the navigation API spec out of WICG and into the HTML Standard. The navigation API move will also happen in October.

Specification

The following sections of the HTML Standard:

• "Loading web pages" excluding "Security infrastructure for Window, WindowProxy, and Location objects", "The WindowProxy exotic object", "Origin-keyed agent clusters", "Cross-origin opener policies", "Cross-origin embedder policies" (upcoming rewrite version has the same scope)
• All contents of https://wicg.github.io/navigation-api/, which will be part of HTML's "Loading web pages" section in October.
• "The iframe element"
• "The a element"
• "The base element"
• "Links" excepting the "Link types" subsection

Optional parts of the standard, which fit with the theme but I would be happy to remove from the proposal if reviewers would prefer that:

• "The embed element" and "The object element": similar to iframes, but less used and more legacy, and their spec is more of a mess because of it.
• The OKAC, COOP, and COEP sections of "Loading web pages": these are new-ish headers, mostly focused around security. I think getting interop on them is good, but I don't see a lot of evidence of developer pain based on these headers' current interop status, so we could omit them.
• The Window/WindowProxy/Location sections of Loading web pages: these sections are really about the subtleties of JavaScript bindings, and not really about history or navigation. Again, interop is good, but the existing interop bugs about these sections are not known to be a huge source of developer pain. (Note that the interop issue of WindowProxy reuse (Rationalize behavior of Window object reuse whatwg/html#3267) is in a different section, and so is included.)

Tests

• html/browsers/, excluding origin/origin-keyed-agent-clusters/, the-windowproxy-exotic-object/, and a few of the tests in the-window-object/ if we don't do the optional components for them

• html/semantics/embedded-content/the-iframe-element/

• html/semantics/text-level-semantics/the-a-element

• html/semantics/links excluding the linktypes/ subdirectory

• html/semantics/document-metadata/the-base-element/

• navigation-api/ (this will be moved into html/browsers/ in ~October)

Optional (corresponding to the above optional spec sections):

• html/semantics/embedded-content/the-embed-element/

• html/semantics/embedded-content/the-object-element/

• html/cross-origin-embedder-policy/

• html/cross-origin-opener-policy/

Note that as part of the spec work mentioned above, over the rest of 2022 we plan to (a) add more coverage to html/browsers/; (b) aspirationally, do a general reorganization and renaming of existing tests of the html/browsers/ directory, since they are some of the oldest tests and thus often their paths don't make any sense or they have useless filenames like 006.html or traverse_the_history_5.html. This should help ensure we're starting from a good baseline, e.g. by fixing any failing-in-all-browsers tests before 2023 starts.

Important note: wpt.fyi is especially fragile for these sorts of tests, often giving incorrect failures. (I.e., failures that don't match what a user sees by loading the corresponding test in the browser on wpt.live.) See, e.g., web-platform-tests/wpt#33590 and web-platform-tests/wpt#23364 . Fixing this would probably need to be a priority for the wpt.fyi infrastructure team, to make this project feasible.

[smaug----]

It is rather surprising to see Navigation API as part of the interop, given that it doesn't even have multiple implementations yet. The spec may still change quite a bit - that just tends to happen with new proposals.

I'm all for the other parts. Could you perhaps split the interop proposal to two pieces, one about the existing navigation and session history pieces and then separate about Navigation API?

[gsnedders]

Could you perhaps split the interop proposal to two pieces, one about the existing navigation and session history pieces and then separate about Navigation API?

I'd strongly second this suggestion; I think plenty of people are gonna view proposals for "support a new feature" very differently to "improve interop of the existing feature", even if both are parts of solving developer pain around the same use-case.

[domenic]

We discussed this a bit internally within Chrome.

We believe the most impact on developer and user needs would come from working on the whole navigation and history space, including the new navigation API as part of that. Indeed, the reason we've been investing so many resources in things like the navigation and session history rewrite, or expanding the base web platform test suite, is to ensure the navigation API has a solid foundation. In other words, developers have told us they want the navigation API; getting interop on the foundations is more for our peace of mind as browser engineers and specification writers, whereas most of the developer-facing impact is in the new API. (And, likely, the user-facing impact, in terms of new experiences enabled.)

That said, I don't fully understand the Interop 2023 process, and it seems like there's some general confusion about it, judging from e.g. #191. If the admins would prefer to split the proposals because that works better with how they're organizing the selection process, then that would be fine. But if the proposals were split, Chrome's position would be to support both or neither, not just one of them. We think general session history and navigation interop is important, because it supports the navigation API; and we think the navigation API by itself would be shaky if it were not accompanied by the foundational work.

Regarding @smaug----'s point of the spec potentially changing due to only being implemented in one place so far, I understand that's part of the Interop 202N process. From what I understand, Interop 2022 included several proposals (e.g. cascade layers, new viewport units) with zero implementations as of December 2021, and several more (e.g. subgrid, color spaces) with one implementation. Updating the specs and tests in response to feedback would be a top priority for our team, were this proposal accepted, just like it has been for the layout team on their 2022 items.

[jgraham]

So in general my understanding is that proposals are going to be treated as atomic and seperate; that is if you believe there's no merit in the cleanup work without the new API (or vice versa), a single proposal is the correct way to do that. We don't have a concept of linked proposals (although probably the final setting of positions will be somewhat iterative, so it's de-facto possible to achieve that).

However it strikes me that insisting on grouping together items which have different levels of support is a strategically poor choice. To use this proposal and an example, and assuming it was split into a "cleanup" proposal and a "new API" proposal, it seems like there are several possible outcomes:

• Everyone supports both proposals. They're grouped together later in the process into a single focus area, and everything turns out just as it would have done for a single proposal, with a little overhead from splitting and rejoining.
• One proposal gets support and the other gets objections. In this case we get to adopt one proposal, and make and measure incremental progress in 2023. Assuming the objections are based on priority decisions rather than fundamental problems, the other proposal is likely to be accepted for Interop 2024.
• Both proposals get objections. In this case it's no different to the single proposal case.

So by inisting on submitting a single proposal rather than splitting, you're removing an avenue toward incremental progress, and making it more likely that none of this work is included in Interop 2023. That makes sense if you believe that all the user value comes from shipping new APIs, but I at least don't think that's well supported: bugs that cause site breakage in shipping APIs cause far more user problems in the short term than the lack of new features. In the case of navigation in particular, the breakage can be severe (failure to load resources, race conditions, etc.) and so fixing these longstanding differences on the platform seems like a clear win for users and authors.

That of course doesn't affect what Chrome can do: if it works for your development process to tackle the entire content of the current proposal in 2023 that's fine. Indeed, if there are lingering concerns about the status of the Navigation API spec, the act of working on an implementation is likely to shake out any remaining issues, and make any future proposal more likely to be accepted.

[domenic]

Thanks all for the advice. Given that, I've split this into two proposals, #199 and #200.