[overlays] Improvements overlay system

[tlouisse]

Context/background

Terminology

In our system we have the so called ‘global’ and ‘local’ overlays. This terminology refers to the connection point in dom: global overlays are attached to the body, local overlays are connected next to their invoker.

Current setup



In our current setup, there is a coupling with the positioning as well: local overlays are positioned relative to their invoker node, global overlays are positioned relative to the viewport. 


Small sidenote: This feels like an arbitrary choice, because we face the same issues with local (Popper) overlays: we experienced this with Storybook demos, which need a global style override to work: https://github.com/ing-bank/lion/blob/poc/globalOverlayMoreCapable/.storybook/preview-head.html



Our current setup allows for a declarative api, similar to the <dialog> spec: web components can be used with content projection for the overlay contents.
This also allows for a single api that makes it possible to switch from a local to a global perspective. For example: a datepicker can be a bottom sheet/dialog on mobile and a dropdown/popover on desktop.




Challenges with global overlays:

1. Style inheritance:

• 1a. CSS variable Style inheritance:
  With a local overlay or <dialog>, local styling context is respected. Css variables automatically inherit. If we want to understand the local context, we need to ‘teleport’ these values. See: [Feature Request] Allow usage of CSS variables with global overlays #744
• 1b. CSS selectors based on surrounding dom structure:
  Adding styles that only apply to the content is cumbersome. With our current setup, it would be possible to provide a style tag in the content of the overlay (so they are moved along to the body), but this feels hacky and can lead to potential clashes (this issue is closely related to issue 1). See again: https://github.com/ing-bank/lion/tree/poc/globalOverlayMoreCapable 
In this poc, host attributes are also teleported, because we need to make sure selectors like ‘:host([my-option]) .my-global-overlay {}’ would still work (.my-global-overlay being the element moved to the body.
  
If we want to cover for all edge cases, we actually should move along the complete html structure of the shadowRoot that contains the overlay(and hide irrelevant parts), to also make css selectors like ‘:host > .my-wrapper > .my-global-overlay’ work (again, .my-global-overlay is the relevant part here).

2. Multi level content Projection:
   It does not take into account multiple levels of content projection. Currently, it just moves the contentNode (which contains its own slotted content usually.) But what about the following edge case:



<my-context>
  
  #shadow-root
  
  <my-global-overlay>. <!— this is the contentNode put in body —>
     
    #shadow-root
     
    <slot name=“overlay-slot”></slot>
     
    <div slot=“overlay-slot”><slot name=“context-slot”></slot></div>
  
  </my-global-overlay>

</my-context>


A user would do:

<my-context><div slot=“context-slot”/></my-context>


In a local context, this works fine. In a global context, projected content would need to be traced back to the ‘actual host’. Theoretically, this can go multiple levels up.

3. Compatibility with multiple template rendering engines:
   Moving content nodes rendered with lit to the body works fine, because lit internally tracks references to dom nodes. For different rendering systems, this might not be the case.
   This is only a problem on the OverlayMixin level (the Overlaycontroller just expects node references). It can be mitigated by allowing a function returning a node reference in the OverlayMixin that could be used instead of content projection. The contents in this node reference can be updated (a.k.a. rerendered) at any point in time by the end user.
4. Accessibility does not work out of the box until AOM:
   Some aria connections will be lost. For instance, when moving the listbox of a combobox to the body, the input will be kept locally and the listbox lost connection to role=combobox. This again could be mitigated by copying the shadowRoot and hide all that is not the overlay via sr-only
5. Events:
   Events should be retargeted as if they were sent in the local context. (something we already do in the FormRegistrationPortalMixin)
   For modal overlays, this also hover events outside the overlay should be blocked (via an (invisible) backdrop?)

Overlays improvements:

• Shadow dom Encapsulation for global overlays:
  The global context has no shadow dom encapsulation. Hence, It’s only possible to move ‘light dom’ (for instance a web component that contains shadow dom) to the body. When an overlay has multiple content slots (for instance the GitHub combobox demo that needs to put the text input in the body as well), it is not possible to move a shadow root and add all the slots via content projection. A poc solution for this can be found here: https://github.com/ing-bank/lion/tree/poc/globalOverlayMoreCapable
• Developer Experience:
  Conceptually hard to grasp for Subclassers, since it behaves/is configured differently for projected and non projected content. Also, the arrow positioning needs to be taken into account. When the approach of the POC branch would be used, a subclasser wouldn’t need to be aware of the difference between projected and non projected nodes.
• Hardening the context switching:
  Switching contexts (local to global and vice versa) sometimes causes bugs (interactiveness date picker broken or just not working at all) (=> find out how these configs differ from those in our demos). It needs more unit tests and the not working cases should be added to the demos.

Challenges with local overlays:

So, why do we not just use position:fixed for overlays that need to be positioned relative to the viewport?

If we look at other component sets (Polymer, material web components), position:fixed is used, combined with a ‘local connection point’.

We move the contents to the body for two reasons:

1. Dom context dependence:

• 1a: Stacking context:
  In a local context, we don't control the stacking order; the siblings in the parent tree do.
  The code below illustrates the problem:

<div style="position: absolute; z-index: 2; top: 0;">
  I am on top and I don't care about your 9999
</div>

<div style="position: absolute; z-index: 1; top: 0;">
  <div style="position: absolute; z-index: 9999;">
    I want to be in front of you
  </div>
</div>

(code example copied from https://github.com/giuseppeg/react-layers-manager)

• 1b: Transforms and overflows:
  if an overlay is spawned within a context that applies overflow or transforms, a fixed positioned overlay cannot ‘break out’ of this. The default styling for of the storybooks are an example of a parent that applies overflow:hidden. 
In order to fix this, a developer needs to be able to change the context his component is applied in. See: https://achrafkassioui.com/blog/position-fixed-and-CSS-transforms/

2. Inert/blocking overlays a11y for browsers not supporting aria-modal:
   For modal overlays in local context, ‘aria-hidden=true’ needs to be applied to all siblings, for the complete path up to the body. Central management for this is needed when multiple dialogs are opened and closed

Possible solutions:

1. Use local approach ‘position:fixed’
   It would solve all of above issues and would simplify our code a lot.
The sacrifice: in some cases, the context (the containing application) needs to be adjusted. This means the feature containing the overlay is less self contained and we shift responsibility to the end user to change his surrounding environment sometimes. In some cases it might not even be possible to do this.
 For issue 7, we are on the safe side here.
2. Continue with the global approach and fix all challenges for global overlays listed above
   Implement all proposed solutions as suggested in the poc and the issue created by Mathieu. This is the direction we already took and invested in. Most problems can technically be solved. It will need a few iterations to become bug free, but not impossible. Only thing that remains a bit uncertain is point 7. Haven’t looked into that enough.
3. Go with <dialog> for forwards compatibility
   For browsers supporting this, everything works fine out of the box. For non supported browsers, we could provide a compatibility layer/shim (that is basically solution 2, which we would need to build anyways). For Chromium browsers (most browsers, Electron apps etc.) we will have a clean and performant (compatibility layer would only be loaded when needed) solution.

   Risks:

• Firefox has the feature behind a flag for a long time already. Will they ever ship it by default? Will Safari also follow?
• keep in mind that we will always need strategy 1 and/or 2 as a backup.


Note that combinations of the solutions above are also possible. This is for instance the approach Popper takes: go with 1 as the main strategy, but allow for 2 as a fallback (although I bet all edge cases described above would not be handled by Popper, this is something we could do).

Extra: some considerations for using global vs local

These considerations might help in choosing one of the possible solutions above.
With the right amount of effort, everything could be fixed with the global implementation. The dom context dependence issues as described above though, are quite impossible to solve (unless we change the parent application).


Why would a local overlay still be an attractive solution, then?



dynamically changing local context

When the global overlay should work as if it was implemented via the c.q. all ‘local context’ would be respected, all changes to that local context should be captured and transferred to the global context. 

One should think of:

• a change in the css variables (affecting both shadow and light dom)
• a change in the parent dom (for instance an host attribute, a class or just a changing dom structure) could affect the styles in the light dom of the overlay
• above points get way more complicated when content projected nodes are not owned by the parent shadow root, but several shadow roots up (see ‘multi level content projection’ for how this happens)
  



performance

• smoothness when scrolling and resizing pages is more expensive for overlays that are positioned globally (this is mainly relevant for elements that are positioned next to their invoker instead of the viewport).

• bandwidth: everything local (or using ) is way easier to implement and requires a lot less code. Thus less code sent over the line
• first paint: before the global overlay is painted in the body, quite some ‘context computations’ need to be done (it should be possible for a developer to turn these off however).
• ‘watching/observing’ local context as described above will be expensive

[MathieuPuech]

Issues with global placement with projected node are:

• events dispatched from the content of the dialog go in body instead of the dialog parent
• css variables applied on the dialog are not applied in the content

Here is an example of dialog made with lion with local placement as default: https://codesandbox.io/s/lion-dialog-rp7e3?file=/src/index.js

Current lion dialog implementation does not allow to have nested dialog with local placement (when closing the second dialog, they are both closed). But I don't think it's an issue because using nested dialog is generally a bad UX.

The local placement has some issues with styling applied to the parent of the dialog, some bugs can sometimes be solved by the user of the overlay (see here). But you can maybe not solve all issues.

<dialog> could be a solution for browsers supporting it but the polyfill of dialog for other browser use position: fixed if I remember. It will result to the issues we already have.

We could wait for this specification but it could take some time.

[tlouisse]

Those are great additions, thanks @MathieuPuech. The event indeed is something we also encounter in the select-rich: when children are registered (and the select is displayed as a global overlay), their events are proxied to the original local context. We could make this behavior more generic.

The top layer research (whatwg/html#4633) was an interesting read and inspiration 👍
I restructured the original post a bit and added some considerations found in these discussions

[jorenbroekema]

Hmmm I'm having a tough time with some of these, not really following what the problems are and whether they are truly bugs or just minor inconveniences.

For me it would really help if we could split this into smaller discussions, because with such a high level overview it's really tough for me to think about how we can move forward with a concrete plan. And we should distinguish between high prio and low prio things.

[tlouisse]

Thanks for the response @jorenbroekema

I can imagine some of the issues are hard to follow when you didn't encounter them yourself.
For instance I encountered some of them when building the combobox github example (the global overlay cannot be used their, only by overriding the Popper css with !important and creating a 'viewport positioned' overlay with position:fixed).
I also encountered the multi level content projection issue in our extension layer.
Maybe I can make a sandboxed example that shows all the flaws described above. Let me know if that would help you or you just need more clarification on some.

Whether you call something a bug or minor inconvenience should actually be the outcome of this discussion. In general I would say that if there is no proper workaround (like in the examples mentioned in paragraph above) it's a bug.
If our goal is to have a Developer Experience similar to using a native <dialog> (so users can use css variables, dom selectors based on local dom etc. etc.), then all of the above are bugs when we move overlay contents to the body.

First we need to find out a main strategy. The question wrt global dialogs is:

1. do we go for a <dialog> like user experience (understanding local context) with position:fixed?
2. do we go for a <dialog> like user experience (understanding local context) by analyzing local context and move/copy that along to the body?

Another question is whether we use the actual <dialog> element when the browser supports it (we have to build the above anyways, so it would serve as an enhancement layer for performance)

A possible outcome could be:
We go for 1 as our main approach, but when the surrounding context has stacking order, transform or overflow problems, we move to the body This escape to the body does however come with limitations (no css variables etc. etc.). Those would be the so called 'minor inconveniences' then.

Once we know the main strategy, we can make separate issues and know the direction for solving each of these issues.

[jorenbroekema]

Yeah some sandboxed examples would really help indeed especially for the stacking order problem and the CSS issues, as well as the multi-level content projection issue..

An example of what is in my opinion an inconvenience: In order to style contentNode of a global overlay, it is best to make your content node a webcomponent so you can encapsulate styling. Very "common" issue with a reasonable solution I would say.

[tlouisse]

True, it's a reasonable solution, but when I tried to create the github combobox, this approach was not sufficient. I will try to reflect this in the sandbox

[tlouisse]

Game changer: <dialog> is supported for all evergreen browsers 🎉 : https://caniuse.com/dialog

That means we can always connect it locally and render it to the top layer. This would highly simplify code, have great performance and tackle all challenges listed above with zero tradeoffs.

[tlouisse]

See: #1712

[RogierdeRuijter]

The popover api has landed in all evergreen browsers (firefox behind a flag) https://caniuse.com/?search=popover this can also be used to simplify the overlay system.