Screen orientation change events as a special kind of user activation

[marcoscaceres]

A common use case on video sites is:

1. A user starts playing a video on their phone in portrait mode.
2. After a few seconds, the user decides they are enjoying the video... so they rotate their phone into landscape mode.
3. Rotating the device triggers a "user initiated" orientation change event.
4. The site can then call .requestFullscreen() and have such orientation changes be classified as "triggered by user activation".

HTML currently specifies "activation triggering input events", but excludes orientation change events as counting as having "transient activation".

The Screen Orientation spec had some hacky text that would magically label these specific orientation events as having been initiated by the user. And the Fullscreen API was depending on that. However, it would be good to have a generic mechanism for labelling these Events properly and having them participate in the user activation model (i.e., they are classified as being part of "activation triggering input events").

[marcoscaceres]

Perhaps a simple solution might be to add a isUserInitiated (bool) either as IDL attribute or internal slot on the Event interface. Then HTML can check for that being true to count as transient activation.

@annevk, @domenic, @mustaqahmed, @michaelwasserman thoughts?

[domenic]

How would that be different from isTrusted?

[marcoscaceres]

The problem is calling screen.orientation.lock(x) can potentially trigger an orientation change.

So, if you have a PWA that is presented in full screen via web manifest (not via .requestFullscreen()), the user agent can't distinguish between a user-initiated orientation change and a screen orientation API initiated orientation change.

(I know you are not proposing this, but) making such events not trusted wouldn't be right, as they are user agent generated events.

[domenic]

I'm hesitant about exposing this as an API until we've done more of a survey of use cases and other places where it might apply, but exposing it as an internal slot seems reasonable. Then we only have to worry about getting it right on specific places where callers need to check it.

Some interesting related notes:

• element.click() already breaks the contract of "if it's user-agent generated, we set isTrusted to true". Although this case is borderline (is it really trusted, if the "user agent code" is basically a wrapper function for dispatchEvent()??), we generally regret it. So indeed, probably following that precedent is not good. Maybe if we'd had something like what you're proposing here, we could have avoided that.

• The navigation API (being merged into HTML soon, I promise!) has a somewhat-similar concept, but I guess it is not overlapping in the end. For its use cases, isTrusted suffices: https://wicg.github.io/navigation-api/#new-user-navigation-involvement

[domenic]

For the internal slot route, we could even be very specific for now, and just have something like "is a user-initiated orientation change event" or "is a special activation triggering event".

[marcoscaceres]

Actually, .isTrusted could work if screen.orientation.lock() consumes the user activation, because then there would be no opportunity to call .requestFullscreen() or any other API gated on user activation.

However, there is some concern that changing the Screen Orientation spec might not be web compatible: w3c/screen-orientation#218

But .isTrusted does provide a really elegant solution if we can change the orientation spec.

[marcoscaceres]

Actually, yeah, thinking about what you said above @domenic... a very specific slot might make most sense because we only have one very specific use case: changing orientation should allow entering/exiting full screen.

It would be bad if changing the orientation would allow, for instance, a popup window, initiate a payment request, initiate a web share, etc.

[michaelwasserman]

A specific internal slot seems like the most appropriate option presented (but I'm still a relative novice here).

[smaug----]

element.click() is a horrible special case (like is dispatching click events from JS), because of legacy reasons. Effectively click handling doesn't care about isTrusted.

It would be quite confusing if UA dispatched event had isTrusted sometimes true, sometimes false. isTrusted is supposed to mean whether UA dispatched it or it was dispatched from JS. That behavior is even documented in the DOM spec.

[marcoscaceres]

Agree, let's put the .isTrusted aside and figure out where to label this.

The other problem is that even though it originates from an event, this needs to available at the window-wide (same as transient activation).

That is, the UA should be to check:

if "is a user-initiated orientation change event" OR "has transient activation" then ... enter fullscreen; otherwise reject.

But "is a user-initiated orientation change event" has to expire with the same timing model as transient activation (and probably shouldn't count as sticky activation).

[domenic]

Although I can see how orientation change allowing window.open() or PaymentRequest is weird, I will say that this will all be a lot easier if you reuse the existing user activation model. If you're willing to accept that consequence, then the spec modification is simply:

• Add the internal slot to such orientation change events
• Add a bullet point to https://html.spec.whatwg.org/#activation-triggering-input-event making such events activation-triggering.

If you want to create a second thing, which is like user activation but not really, that's much more work. You would need to define something like:

• Each window has a last orientation change timestamp
• A new algorithm similar to https://html.spec.whatwg.org/#activation-notification which updates the last orientation change timestamp whenever the user changes orientation
• A new predicate, e.g. "has recently changed orientation", which compares the Window's last orientation change timestamp with the current time and the transient activation duration. (Or with a separate duration, if we want to allow them to vary independently?)
• Maybe a separate consumption algorithm, similar to https://html.spec.whatwg.org/#consume-user-activation, that resets the last orientation change timestamp for a window and its descendants. (This would be necessary if you want to prevent repeated fullscreening and un-fullscreening and then fullscreening again, in response to a single orientation change.)

[marcoscaceres]

Ok, I'm willing try the easy route and see if folks object. As already mentioned above, there are consequences, as some are a bit unexpected (specially around orientation change... particularly on desktop platforms, where screen orientation is usually changed through system settings instead of merely rotating one's hand). It's a bit weird, but so long as activation consuming APIs do the right thing, then it may be ok.

@michaelwasserman how much "orientation specific" handling does Chromium have to handle the full screen specific case? In particular, how is timing handled in Chrome as @domenic describes in the second solution above handing Fullscreen + orientation change?

@smaug---- or @EdgarChen, would you be amicable to considering orientation "change" events as an "activation triggered input event"?

[smaug----]

I think we should reuse user activation model as Domenic suggested, so considering some "change" events as user input sounds reasonable - assuming it is implementable. Do the OS level APIs let one to know how the orientation change was triggered?
If not, one possible solution might be to check in https://w3c.github.io/screen-orientation/#dfn-screen-orientation-change-steps whether there are orientationPendingPromise objects around, and if so, treat the change as non-UA triggered.

[michaelwasserman]

Granting transient user activation on orientation changes seems convenient, if excessive.

I think Chrome currently grants fullscreen during execution of screen.orientation change event listeners, not for a fixed duration after the event.

[mustaqahmed]

Granting transient user activation on orientation changes seems convenient, if excessive.

Thanks for calling out the "excessive" point. If the user changes the device orientation, it doesn't necessarily imply an interaction with the page, and that bring up a few challenges IMHO. For example, switching to landscape mode for convenience (like getting a wider view on the phone) is not uncommon. It seems pretty bad to me if a page can suddenly a open popup or start playing audio right after this event. Even worse would be a malicious ad iframe getting the ability to call elem.focus() or navigate away to another site.

[smaug----]

That is a good point. Especially when using mobile devices I'd expect pages to get quite a few accidental change events.

[domenic]

It's definitely a tradeoff between convenience and excessiveness, as I outlined in #8490 (comment). I come down on the convenience side, but if Gecko and Chromium would prefer defining a new concept to ensure we avoid any extra transient activations, that's probably fine...

[smaug----]

Could we tweak the existing model so that transient user activation is effectively a bitmap. In normal cases everything would be allowed (all the flags set), but in this case only one flag - fullscreen, was enabled.

[domenic]

I can't see how that would really work. You still need to store separate timestamps for the different events, so it ends up seeming equivalent to the model I proposed.