Previously: Daniel Libby, Stephanie Stimac
| Name | Link |
|---|---|
| Device Posture API | Explainer, Working Draft Spec |
| Multi-Screen Window Placement API | Explainer, Draft Community Group Report |
| Visual Viewport API | Draft CSSOM View Spec, MDN |
Web developers targeting foldable devices want to be able to effectively lay out the content in a window that spans multiple displays or across the fold of device with a foldable screen. However, the web platform does not yet provide the necessary primitives for building layouts that are optimized for foldable experiences. Developers may be able to solve this by taking a hard dependency on a specific device hardware parameters - an approach that is fragile, not scalable, and requires work duplication for each new device.
More specific challenges we've heard from our internal product teams that were exploring building experiences for this emerging classes of devices include:
- Hardware differences: Devices could be seamless (e.g. Samsung Galaxy Fold, Asus Zenbook Fold) or have a seam (e.g. Microsoft Surface Duo 2, Asus Zenbook Duo or ZTE Axon M). Developers might want to take it into account or intentionally ignore depending on scenario;
- Folding capabilities, state: the fold area could be safe or unsafe region to present content;
- Future-proofing: Ideally developers would want a somewhat stable way to target this class of devices without having to rely on specific device hardware parameters.
Before discussing the solution proposal - let's overview existing proposals that are relevant and applicable to the problem space. As matter of principle we should generally avoid creating redundant concepts if we can reuse existing platform APIs and capabilities.
-
Device Posture API is exposing the physical position of the device, called posture. In foldable devices the content can be consumed and browsed even if the device is not flat and this allow the developer to create a differentiated UI depending on the posture state. Developers targeting foldable devices will very likely use the Device Posture API and the Viewport Segments API to build their experiences.
-
Presentation API is solving the problem of a effective use of a secondary screen and likely won't attempt to solve challenges outlined above that are specific to devices where a window can span separate physical displays. This would likely still be a separate problem for foldables.
-
Screen Enumeration API Explainer provides information about the physical screen configuration. Web developers might be able to leverage that on foldables, but would need to do extra effort to correlate that information with window parameters. Some concrete examples on why a special purpose API might be useful in addition to Screen Enumeration:
- Getting adjacency information about spanning window regions to lay out content in several areas in logical way for a device;
- Getting inner window dimensions that account for application frame, OS UI elements, etc.
-
Window Placement API Explainer is useful in multi-window scenarios on multiple screen devices, but does not target scenarios in which the hosting application (i.e. browser) has a single window which spans multiple displays. In this case, the developer may not wish to open new windows - just hints to help lay out things properly and take advantage of the physical partitioning of the available layout space.
Additionally, while not a solution in the same sense, a "[css-media-queries] Foldables support and enablement" issue discusses the problem space and outlines some details and touches upon outlined issues.
A summary of the concepts from the other proposals:
- Display - the logical representation of an physical monitor.
- Screen - the aggregate 2D space occupied by all the connected displays.
We propose a new concept of Viewport Segments, which represent the regions of the window that reside on separate (adjacent) displays or across a seamless folding screen. Viewport Segment dimensions are expressed in CSS pixels and will be exposed via a JavaScript API that allows developers to enumerate segments where logically separate pieces of content can be placed.
This proposal is primarily aimed at reactive scenarios, where an application wants to take advantage of the fact that it spans multiple displays, by virtue of the user/window manager placing it in that state. It is not designed for scenarios of proactively placing content in a separate top-level browsing context on the various displays available (this would fall under the Window Placement API or Presentation API). Note that given the Screen Enumeration API and existing primitives on the Web, it is possible to write JavaScript code that intersects the rectangles of the Display and window, while taking into account devicePixelRatio in order to compute the interesting layout regions of a window spanned across displays. However this may not correctly handle corner cases of future device form factors, and thus this proposal tries to centralize access to "here are the interesting parts of the screen a developer can target or consider for presenting content" as a practical starting point.
interface VisualViewport : EventTarget {
readonly attribute FrozenArray<DOMRect> segments;
}
The value returned from querying visualViewport.segments will be an array of DOMRects and based on the data returned for each "viewport segment", developers will be able to infer the number of hinges available as well as the hinge orientation. When visualViewport.segments is queried on a device with only a single segment, this will return null. This isn't exposed in this case because this information is redundant with other fields on VisualViewport. This also avoids future compatibility issues so that authors don't start using visualViewport.segments[0] to target single-screen devices.
A user may at any point take the browser window out of spanning mode and place it on one of the screens or vice-versa, in those cases the window resize event will fire and authors can query and get the number of available viewport segments.
This proposal doesn't aim to substitute existing APIs — the proposed development model can be summarized as requesting current viewport segments on interesting events and adjusting to the new presentation environment. There are no additional lifecycle proposals - the viewport segments are immutable and developers would request them upon common sense events (e.g. orientationchange, resize). It also doesn't suggest how developers would use viewport segments to position, scale and orient content - in practical explorations developers used viewport segments to select the best declarative layout, not to modify layouts in script, but either would be possible.
No new security considerations have been reported on this specification.
The Viewport Segments API exposes segments determined from the hardware when the device is used in certain postures (see Device Posture API for more details). Segments are in CSS pixels and relative to the viewport, they do not expose any global position in screen coordinates.
Just like the viewport size, segments could be specific to a given device model which means they could be used for fingerprinting. If this API can be used simultaneously in different window contexts on the same device it may be possible for that code to correlate the user across those two contexts, creating unanticipated tracking mechanisms. One example would be to determine if two users are not the same. Another would be to use the viewport segments sizes however, in practice as more and more foldable devices gets released the value of this information will be diminished (especially considering that you could use the same logic on the viewport size itself).
As of today, segments will return null when called from within an iframe context but the CSS API will be available.
If segments becomes available to iframes, it would bring a new privacy issue. Except when the iframe is fullscreen, segments values need to be adjusted to
be useful in the iframe context. If so they could be used to potentially find the iframe position in the main frame. But this is doable today, covered in
the Intersection Observer Privacy section and the conclusion is that the privacy implications are unclear.
Two mitigation strategies could be added:
- Restrict the viewport segments to same origin iframes
- Restrict the viewport segments to fullscreen iframes (or iframes spanning the whole viewport)
- Secure Context: the
segmentsare only available within a secure context. - Visibility state: the
segmentsproperty may return stale values and is only updated while the document's visibility state is "visible".
Let's take a look at a few practical examples of the scenarios above and how viewport segments would allow to resolve them for better user experience. In each case we'll start with some existing scenario and complicate it to provide opportunity to apply the proposal.
const segments = window.visualViewport.segments;
if (segments && segments.length > 1) {
// now we know the device is a foldable
// and we can update CSS classes in our layout as appropriate
document.body.classList.add('is-foldable');
document.querySelector('.map').classList.add('flex-one-half');
document.querySelector('.locations-list').classList.add('flex-one-half');
}We can also use the @horizontal-viewport-segments and @vertical-viewport-segments media queries to change the layout and style of a website depending on the way a device is oriented.
Additionally, there are a number of new environment variables that can be used to place and position content on dual screen devices.
window.onresize = function() {
const segments = window.visualViewport.segments;
if (segments && segments.length > 1) {
// Make changes two split content into the segments.
} else {
// Reset state to single viewport (normal responsive layout).
}
}