Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposing new CSS primitives to enable great web experiences on foldable & dual-screen devices #4736

Open
zouhir opened this issue Feb 4, 2020 · 10 comments

Comments

@zouhir
Copy link

@zouhir zouhir commented Feb 4, 2020

My colleague @dlibby- and I have published an explainer on MicrosoftEdge's Explainers repo where we have proposed a new JavaScript API and a CSS media query with a set of pre-defined env() variables. Both aim to enable developers to lay content out effectively in a window that spans multiple displays, and more specifically for dual screen devices.

We wanted to communicate our thought process behind why are we proposing the new CSS primitives and collaborate with the working group to help web developers continue to build great responsive experiences on the web for every device that runs a browser.

The flexible form factor trend

The trend started in 2019 when Samsung, Huawei, Asus and others released their foldable & dual screen devices, and in 2020 Microsoft will be adding 2 devices, Surface Neo and Surface Duo to this flexible device category.

Why are we proposing new CSS primitives?

When the browser window is spanning across the device fold, informing developers about the fold orientation and display boundaries can help them design great experiences. Here’s a quick overview of possible patterns and opportunities for the web platform when CSS can provide the necessary information on this class of devices:

1. Bringing wider-screens UI patterns to smaller portable devices & Creating enlightened UIs on the web

Due to screen size limitation, traditional portable touch-screen devices rely a lot on “stacking views” where clicking on an email in your inbox list for example will cause that whole inbox list view to be replaced with the selected email content view. This behavior usually creates additional navigation steps while wider screen devices have a more natural configuration where the email inbox list view and email content view are side-by-side.

When the browser is spanning across the 2 screens on a dual-screen device developers should be able take advantage of the wider-viewing area and also snap to the natural boundary of each screen.

master-detail

We also strongly believe that re-creating patterns that make sense is not the only PRO here - this device category with 2 displays and flexible hinge can offer unique opportunities for developers and designers to create new experiences as long as the web platform provides necessary information (eg. screen boundaries, fold orientation).

companion-pane

2. Easily improve existing sites & UI components

Developers might not want to introduce major UI changes for this class of devices and would just want to simply move some components around. In the example below, it makes more sense for the modal dialogue to avoid the device fold, whether the foldable device is seamless or has a seam; and we want CSS to offer the capability to match this device class and allow developers to progressively enhance their site.

avoid-seam-dialog-neo

Operating systems & Native app platforms status

Microsoft has released some information about OS & native app platform APIs and SDKs.

We’d like to have a standard declarative web API browser vendors can implement so website and PWAs can match native app platforms capabilities.

Proposed CSS primitives and general design principles

The 'spanning' CSS media feature

A simple CSS media feature to test whether the browser window is spanning across multiple displays on a dual-screen device.

Frame 21 (1)

The spanning media feature value can be one of the following keywords:

  • single-fold-vertical: This value matches when the layout viewport is spanning a single fold (two screens) and the fold posture is vertical.

  • single-fold-horizontal: This value matches when the layout viewport is spanning a single fold (two screens) and the fold posture is horizontal.

  • none: This value describes the state of when the browser window is not in spanning mode.

Device fold CSS environment env() variables

We propose the addition of 4 pre-defined CSS environment variables fold-top, fold-left, fold-width and fold-height. They represent the fold's (device hinge) rects and web developers can utilize those variables to calculate each screen segment size at both landscape and portrait orientations.

Frame 10 (3)

Some notes on our general design principles

  1. Introduce as little new API surface area as possible

Our first attempt tried to find a way to infer the hinge orientation using the existing orientation media feature. However, we quickly found that this is something we can’t generalize; a device can be in landscape orientation but has a vertical fold (folds like a book) another device in landscape may have horizontal fold (like a laptop).

  1. Future proofing

It’s much simpler to design a JavaScript API that returns an array for DOMRects and have it scale for a device with n number of screen; the reason is in JavaScript mappings are more direct, we have concept of for loops, conditionals and arithmetic operations. These can be combined to enable the developer to react to arbitrary topologies.
For CSS, we decided to scope our solution to this new class of device that has specific design principles around how the displays are utilized by an application. However, we believe the new media feature can be extended with new values that match future form factors down the line.

Final thoughts

We'd like to hear your thoughts and feedback, so please feel free to use this GitHub thread as a discussion forum or open issues for us on our Edge Explainers GitHub repo.

Additional material you might be interested in looking at:

@prateekbh

This comment has been minimized.

Copy link

@prateekbh prateekbh commented Feb 5, 2020

I have 2 question:

  1. what would be a potential use case of env(fold-width) and env(fold-height)? also isn't 100vh === env(fold-height)?
  2. Would top:0; right:0 === top: env(fold-top); right: env(fold-left)
@atanassov atanassov assigned atanassov and unassigned atanassov Feb 5, 2020
@atanassov atanassov added the Agenda+ label Feb 5, 2020
@fantasai

This comment has been minimized.

Copy link
Collaborator

@fantasai fantasai commented Feb 6, 2020

I'm a little concerned about extending this out in the future, and I'm wondering if maybe we should be splitting this out a bit more e.g.

display-span-x: <integer[1,inf]>
display-span-y: <integer[1,inf]>

We might not be able to return all the fold information for 5 folds via an environment variable right now, but at least the media query can be a little more accurate and can extend more gracefully into the future possibilities of N panels.

@css-meeting-bot

This comment has been minimized.

Copy link
Member

@css-meeting-bot css-meeting-bot commented Feb 6, 2020

The CSS Working Group just discussed Proposing new CSS primitives to enable great web experiences on foldable & dual-screen devices.

The full IRC log of that discussion <dael> Topic: Proposing new CSS primitives to enable great web experiences on foldable & dual-screen devices
<dael> github: https://github.com//issues/4736
<dael> dlibby-: As maybe you've seen MS has announced 2 new dual screen devices.
<dael> dlibby-: Given these are cross variety of platforms and browsers and web would be great. CSS is great for layouts. Been bandying about ways to expose CSS for this so they can control how webcontent is in relation to the hinge.
<dael> dlibby-: Had some principles about don't want to expose unnecessary new capabilities. Trying to be cognizent that these devices aren't comprehencive of future. Don't want to shut door on future form factors.
<dael> dlibby-: It's a new media feature called 'spanning' Intent is to desc when viewport is spanning several screens. Depends on how viewport is positioned.
<dael> dlibby-: Need to understand where gap/hinge is in terms of css coord. Proposed 4 env variables to desc where is the fold, width and height of fold
<florian> q+
<astearns> ack florian
<dael> dlibby-: Excited to hear feedback, if this sounds reasonable, any drawbacks. Definitely open to changing based on collective experience
<astearns> q+
<dael> florian: Always a challange for a new MQ for new device. Categorizing devices doesn't stand as time passes. I'm happy to see this proposal is trying to test a relevant aspect of what's going on. Means we're on the right path. A little concerned about env variable. When it comes to sizing I think this is likely to overlap the unsolved viewport units conversation and which parts have and don't have scrollbar and keyboard
<dael> florian: All the struggle with vh/vw seem to overlap here. I don't have a solution. Proplem space seems reasonable.
<dael> dlibby-: Makes sense. Aware of some interaction with things like vch unit to reference visual viewport. I would agree there's rationalization to be had there.
<dael> dlibby-: Being desc with viewport units I don't think it sits how we've been thinking about it. Not sure if you're jsut noting tie ins
<dael> florian: Similar issue, not nec. units. If you want to size something to 80% after folding, what 80% are you talking about? Probles are same as vh/vw even if not same units.
<dael> jensimmons: Question: Haven't thought a lot abotu devices with hinge. Is mental model for anyone designing for screen simply that there's a wider screen with a web window like we've known for years and then suddenly it's magically half as wide and that's it? It jumps wide to narrow and narrow to wide?
<dael> jensimmons: Or is it that people think about design of the screen where you have more space then that. WHen you open a browser window it's usually one space and hwen you go wide it goes two up? Are there thing like that?
<dael> jensimmons: Is it simply a different way but similar to responsive or is it more complicated whre you have 2 up b/c new screen?
<Rossen_> https://user-images.githubusercontent.com/5052316/73715033-8a3e5b00-46c7-11ea-8839-af3801c97502.png
<dael> Rossen_: A little of both. Awareness of hinge is important and that's what desc spanning. Spanning is going across hinge in middle. THat allows you to create different experience on that hinge location
<dael> Rossen_: If you look at motivational examples in issue they're pretty well thought through. If there's an email app you can do solumns on left for folder, email titles and then right side is full experience with selected message
<myles> q+
<dael> Rossen_: These are doable but if you don't know where hinge is you have content half on one side and half on the other. That's a simple example. Many others that Zo (sp?) has been working on
<dael> Rossen_: Knowing where hinge si is important
<heycam> q+
<dael> dlibby-: Knowing where hinge is and mapping to it might be useful to developers in many cases
<dael> jensimmons: Some of these things we know so far like where is screen aren't issues. THIs is where hinge is.
<dael> Rossen_: Right. And this is why we're nto defining viewport, but there's something that splits viewport
<fantasai> s/many cases/many cases. Also different OSes treat the gap differently, e.g. some mask content in that area whereas others split the screen apart/
<dael> astearns: Is this prop MQ matching cases where phsycial hardware doesn't have hinge. Thinking like a reader view or something like that where might want to layout differently between left and right page. I think ther answer is this MQ is not for a case like kidle to view on content where content flows from one side to another. This is about having a phsycical gap in screen
<astearns> ack astearns
<florian> q+
<astearns> ack myles
<dael> Rossen_: I dont think we're insisting on physical paradigm. I can see an epub that uses multiple tabs internally to drive that experience using this css mech. It's something we haven't thought about. Good to consider
<dael> myles: It's clear there are use cases for one browser window to span both halves. I've also seen use cases like in explainer with google maps where one half has map and other is locations. For one big browser window we can do that today. 2 panes could be solved with preso API
<plinss> q+
<dael> myles: Your prop is more powerful b/c allows mix both these. Some content aligns to fold and other span both. I haven't seen cases for this mix. Can you desc one?
<dael> dlibby-: I think you said there's single window paradime and you split content. THe other where you lean on preso API to open another full window. Correct?
<dael> myles: What type of content are you encouraging web designers to create with this API. Because with no API web designers could create for this. Presentaion API would encourage 2 different independent. You haven't proposed either of those types. You're proposing a 3rd type that doesn't fall into first two buckets. What type of content are you encouraging that's not in those buckets?
<dael> astearns: Specifically a mixed mode were some content is split but other content spans both?
<dael> Rossen_: Let's consider a mail app with an application bar that spans both screens and then left and right sections for the two screens
<dael> myles: Yeah, I think that answers. A nav bar that spans across the fold.
<astearns> ack heycam
<dael> heycam: A little unclear to me if the model here is the window that spans the two displays. Does it have a strip of pixels which are not rendered because they overlap the folder or is the back buffer for the window is split? Or both?
<dael> dlibby-: Accomodate both. Early MS devices is that they'd have different. One has pixels across the back and the other has a split. In one case your fold-width would be 0 but it's still there
<dael> heycam: Understood.
<dael> heycam: For UI or content you want to span, the toolbar and you've got buttons you want to avoid the fold space so you don't lose half a button- not sure if there's a great way to use env variable to make them avoid fold if you're using flexbox. I gues syou can fallback to script
<dael> Rossen_: What you're saying is exactly what [missed] to avoid having loss of content to avoid having the button fall under the hinge
<dael> Rossen_: If you see fold width it essentially describes that space
<dael> heycam: I'm imaginging a flexbox container that spans the width. I'm not sure how you use nth env variable to make sure the flex items when you get to fold one doesn't go over fold.
<dael> florian: Similar is a 3 column grid where you want 2 and 1 and you don't care which side is the 2 but you don't want a split grid.
<dael> ??: First one is a single flexbox with an arbitrary number of buttons with width. You won't know when buttons reach hinge but it'll help you make 2 columns. Request is a single flex and avoid the gaps
<dael> heycam: Right. THat might be an example of the more general region flow
<astearns> s/??/Zouhir/
<dael> Zouhir: The original demos we experimented with we didn't touch on arbitrary number of childrens avoiding hinge. We did make multiple grid and flexbox where we can snap to the fold nicely. It's a good note to take
<Rossen_> q?
<dael> astearns: Good to add examples of env variables with layout mechanisms
<florian> q-
<fantasai> It sounds like you really need media queries against the fold offsets
<astearns> ack plinss
<fantasai> plinss++
<Rossen_> +1 to plinss
<dael> plinss: Couple points- Lot of work done on folding screens and I'm hesitant to do something in CSS that we'll have to carry around when folding screens might no longer have hinges. However multiple monitors on desktop is comment. Would like to make sure if we solve what we do applies on desktops as well. Food for thought
<dael> dlibby-: Good feedback. Have that in the back of our mind but not concrete
<dael> Rossen_: We've taken steps to make sure that's not obstructed. But yeah multi-monitor is first that came to discussion
<dael> plinss: And there's 4 monitors in a grid where you have vertical and horizontal
<dael> astearns: Thanks for the introduction. I'm sure we'll discuss on the issue and again on a call
@frivoal

This comment has been minimized.

Copy link
Collaborator

@frivoal frivoal commented Feb 6, 2020

I'm wondering if flex and grid need some kind of extension to allow positioning of "hinges" in them , in a way that the flex/grid layout algorithm could take into account (possibly but no necessarily including a gap at that point). In the case of a flex, that would probably behave sort of like a flex-wrap in terms of laying out and sizing items within each piece of the flex, but unlike flex-wrap, the two pieces wouldn't be on different lines, but contiguous to eachother. If we had something like that, it feels that the environment variables being discussed here would be a lot more useful.

Even then, another difficulty with the environment variables is that they give you the position (and size) of the hinge relative to the screen/viewport, but to be able to use them in css for the layout/position/sizing of things that aren't the root element (or absolutely positioned with the root as a containing block), that's not super useful: you'd need to know about whether your parent element / parent formatting context is being cut by a hinge, and if so where in terms that are relative to your layout space.

Without that, it feels like you're going to have to use complicated calc expressions to duplicate the way your ancestor chain has been sized / position in order to figure out where the hinge falls with regards to you.

@henke37

This comment has been minimized.

Copy link

@henke37 henke37 commented Feb 6, 2020

Multiple screens is going to lead to some fun in general with media queries. Nearly all queryable properties can differ depending on the screen being used. It would be... bad... not to allow content to query the capabilities of each screen it is using.

It is hardly unlikely that someone is using a laptop (close to user, high dpi) connected to a beamer (far from user, different aspect ratio). Or just a big TV sized screen.

And then there is the issue of hardware acceleration. Screens aren't equal, but neither are graphics cards. Ain't no rule that all graphics cards in a machine are equally capable. Heck, I bet that there are setups that have multiple candidate graphics cards that share the same screen with a software controlled switch. In fact, I hear that's a common setup on laptops.

@astearns astearns removed the Agenda+ label Feb 6, 2020
@diekus

This comment has been minimized.

Copy link

@diekus diekus commented Feb 7, 2020

Is there any specific reason why the spanning and environmental variables are targeted towards dual screen devices? My main concern is that there should be a way of identifying a foldable independent of if it is seamless or with a seam. This proposal can be useful even on seamless devices (I’m thinking here of Galaxy Fold and other upcoming devices) to adapt the UX based on the state of the device. At the moment we are working in Samsung on an explainer that will shed more light on scenarios and new capabilities and how they would combine with the spanning media feature and fold environmental variables.

Out of curiosity, in a device like the Galaxy Fold or the Motorola Razr, and according to the definition provided in the explainer about ‘spanning’, would it resolve to none? If that’s the case, it might be better to specify the ‘single-fold-vertical’ or ‘single-fold-horizontal’ and appropriate env fold variables to potentially accommodate the content on the web page?

To summarize, my concern is with the ‘multiple screen’ wording that keeps appearing in the explainers, since these concepts are useful for single screen devices as well.

@dlibby-

This comment has been minimized.

Copy link
Contributor

@dlibby- dlibby- commented Feb 7, 2020

@fantasai- That is a great suggestion - your proposed features would indeed make extensibility much easier for authors to reason about, and removes the hurdle of future form factors needing to come up with various individual values to describe available configurations. We'll look into testing the concept and validating with authors that have been keen on trying out these features.

@frivoal- Thanks for the feedback and use case! We had not fully considered the best way to treat containers with children that want to avoid the fold. Many of the design patterns suggested for the Microsoft form factors have focused on, as you mention, a high level splitting of content, or absolutely positioning things around the hinge/fold. We'll take another look and see if the current environment variables can be harnessed or if we need some additional mechanisms to enable these scenarios.

@diekus- The proposal has is not aimed at making the spanning and environment variables exclusive to dual screen devices. I'll go back and re-review the wording to clarify this point.

We're trying to expose the underlying system's description of the semantics of the window. For devices like Galaxy Fold or Motorola Razr, if the window manager reports a fold (I believe the currently discussed abstraction for Android is DisplayFeature), then spanning would resolve accordingly. Additionally the information the system exposes regarding the geometry of the fold would populate the environment variables.

If the window manager doesn't expose this information (i.e. expects apps to just use the available space and not treat it as logically split into content regions), then spanning would resolve to none.

@Dan503

This comment has been minimized.

Copy link

@Dan503 Dan503 commented Feb 7, 2020

The media query would need to be able to handle multi screen setups. Users could have a huge variety of set ups though.

There are sensible 3 screen set ups like this:
Man at desk using 3 computer monitors

All the way up to something ridiculous like this:

monstrous 24 screen computer set up

The users display also might not even be rectangular:

6 screen display, 4 in a square, 1 extra screen either side

The 3 screen set up is sensible to support. Do we want to support the more intense screen set ups as well though?

@dlibby- dlibby- mentioned this issue Feb 7, 2020
1 of 1 task complete
@Crissov

This comment has been minimized.

Copy link
Contributor

@Crissov Crissov commented Feb 8, 2020

fold-top, fold-left, fold-width and fold-height.

It would feel very unCSSy to not have respective bottom and right fold positions as well.


Overall, this issue raises some important points on design constraints, but the actual representation thereof in CSS, MQ or environment variables requires some additional thinking.

Some display mosaics simply represent a fixed segmentation of a common shared planar surface (i. e. the viewport), like an infinite virtual wall. The static edge width around each screen (and thus between any two neighboring screens) may vary, often approaching zero for a seamless appearance, and are important to consider in layout because the wall may either be split at these frames or be covered virtually by the physical bezels, i. e. continue invisibly beneath them.
In some cases, like a detachable mobile second screen that extends the desktop plane, the distance and relative (planar and even spatial) location is even variable.
In other cases, the wall (or its displayed projection, i. e. the viewport) slants by a fixed-angle bracket or by a variable-angle hinge at the joint, where the latter becomes a fold if only 0° and 180° are stable positions. When this knee is smoothed out and hence seamless, it is a bend, otherwise a sharp kink. (All terms up to bikeshedding by native speakers of course.)

For the simplicity of a first level, we may assume that the gutter between all segments is the same and distributed evenly, except we would distinguish horizontal from vertical ones.

We should also note that the classic spread of left and right pages in a printed book or magazine is conceptionally very similar to two adjacent screens. They can also be more complex, e. g. in centerfolds. If media can be folded completely at a joint, the distinction between recto and verso or front and back also becomes important.

Media Queries

  • segmentation-x or horizontal-segments: maximum number of horizontal viewport segments spanned
  • segmentation-y or vertical-segments: maximum number of vertical viewport segments spanned
  • segmentation: none | [vertical | horizontal | both] [continuous | separate]?
  • segmentation-gutter[-x|-y]?: [seam | frame | bezel]# or [invisible | visible | covered]#
  • segmentation-joint[-x|-y]?: [flat | bend | bracket | hinge | fold]# or [straight | [curved | sharp] [static | variable | alternate]?]#

Environment variables

  • env(vertical-seam-width)
  • env(horizontal-seam-width)
  • env(vertical-seam-angle)
  • env(horizontal-seam-angle)

Desktop setups with multiple screens would often have angles between 0° and 45° with
A cubic cave, with (up to four) walls, optional floor and ceiling, would usually have zero-length seam widths and right seam angles.

For curved surfaces (and spheric caves, i. e. physical rooms, or virtual rooms around the user), we might want something similar:

  • env(bend-radius)
  • env(horizontal-bend-radius)
  • env(vertical-bend-radius)
@darktears

This comment has been minimized.

Copy link

@darktears darktears commented Feb 10, 2020

Out of curiosity, in a device like the Galaxy Fold or the Motorola Razr, and according to the definition provided in the explainer about ‘spanning’, would it resolve to none? If that’s the case, it might be better to specify the ‘single-fold-vertical’ or ‘single-fold-horizontal’ and appropriate env fold variables to potentially accommodate the content on the web page?

Something to add to this is that in single screen devices like the Galaxy Z Flip or what Intel showed as the Horseshoe bend concept, the device is partially folded and the fold may or may not be usable (touch not working reliably or image distorted).

So indeed the single-fold-vertical/horizontal properties could be defined even if there is no physical seam because the platform could or should define one arbitrarily. Also the platform/vendor may choose to give the opportunity for the developer (if she/he wants) to provide a split layout (top and bottom of the app being different) because it makes more sense for the website.

To summarize, my concern is with the ‘multiple screen’ wording that keeps appearing in the explainers, since these concepts are useful for single screen devices as well.

I second this as per above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.