Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VirtualKeyboard API - show/hide policy #498

Closed
snianu opened this issue Apr 10, 2020 · 43 comments
Closed

VirtualKeyboard API - show/hide policy #498

snianu opened this issue Apr 10, 2020 · 43 comments

Comments

@snianu
Copy link

@snianu snianu commented Apr 10, 2020

Hello TAG!

I'm requesting a TAG review of VirtualKeyboard Policy API .

The Virtual Keyboard (VK) is the on-screen keyboard used for input in scenarios where a hardware keyboard may not be available. Unlike a hardware keyboard, a VK can adapt its shape to optimize for the expected type of input. Authors have control over the displayed shape of the VK through the inputmode attribute, but have limited control over when the VK is shown or hidden.
We propose a new VirtualKeyboard interface that has APIs to provide authors with more control over when the VK is shown or hidden and also exposes the size of the VK so the page can reflow its content if part of it is occluded by the VK.

Further details:

  • [X ] I have reviewed the TAG's API Design Principles
  • The group where the incubation/design work on this is being done (or is intended to be done in the future): Editing Task Force
  • The group where standardization of this work is intended to be done ("unknown" if not known):
    Editing Task Force
  • Existing major pieces of multi-stakeholder review or discussion of this design: w3c/editing#225
  • Major unresolved issues with or opposition to this design: N/A
  • This work is being funded by: Microsoft

We'd prefer the TAG provide feedback as (please delete all but the desired option):

🐛 open issues in our GitHub repo for each point of feedback

@annevk
Copy link
Member

@annevk annevk commented Apr 20, 2020

This was previously discussed in whatwg/html#4876 and given the last comment there I'm somewhat surprised it's now resurfacing here.

@kenchris
Copy link

@kenchris kenchris commented Apr 20, 2020

@annevk Looking at the explainer, this new behavior is opt-in

@kenchris
Copy link

@kenchris kenchris commented Apr 20, 2020

As this actually seems to touch two different areas - there are even two different explainers - we should probably split this up into two issues

@kenchris
Copy link

@kenchris kenchris commented Apr 21, 2020

I split up part of this into #507

@kenchris kenchris changed the title VirtualKeyboard API VirtualKeyboard API - show/hide policy Apr 21, 2020
@kenchris
Copy link

@kenchris kenchris commented Apr 21, 2020

There are a lot of good use-cases here: whatwg/html#4876 (comment)

I think these would be good to add clearly to the explainer

@torgo torgo removed this from the 2020-04-20-week milestone Apr 21, 2020
@torgo torgo added this to the 2020-04-27-week milestone Apr 21, 2020
@cynthia
Copy link
Member

@cynthia cynthia commented Apr 21, 2020

@snianu Given the feedback from @annevk above, is this proposal a conflict resolution request, or is there a misunderstanding and this review does not touch on the issue mentioned? Would be good to have a clarification.

@snianu
Copy link
Author

@snianu snianu commented Apr 21, 2020

Thank you everyone for all the feedback!
@cynthia @annevk This issue specifically talks about inputMode attribute and this comment was about deprecating inputMode="none"'s special behavior of hiding the VK which we believe is conflating two different states and doesn't cover some of the scenarios mentioned in the explainer. This feature gives explicit control of the VK to the web authors. It also defines an attribute that differentiates between the custom (virtualKeyboardPolicy="manual") and default(virtualKeyboardPolicy="auto") VK behavior. When virtualKeyboardPolicy is "manual", inputMode ="none" 's hiding of the keyboard state is ignored, but the default VK layout is kept intact. Basically, with virtualKeyboardPolicy attribute, the web authors could control the VK behavior without affecting the layout of the VK which would be controlled by the inputMode attribute. These two states (visibility and layout of the VK) are kept separate in this proposal so web authors can use it to facilitate more complicated scenarios on their sites.
The new API can be feature-detected by checking for the existence of the virtualKeyboard property on the navigator object.

@kenchris This section talks about the range-peeking scenario in Excel that sort of merges the two use cases that were mentioned in this comment, but I agree that we should explicitly mention that in the explainer. I'll create an issue in our repo and address that in the explainer.

@annevk
Copy link
Member

@annevk annevk commented Apr 23, 2020

@snianu from what Ryosuke stated it seems unlikely to change however, which is more relevant I'd think.

@hober
Copy link
Contributor

@hober hober commented Apr 23, 2020

(@annevk's referring to this comment of @rniwa's from February.)

@BoCupp-Microsoft
Copy link

@BoCupp-Microsoft BoCupp-Microsoft commented Apr 23, 2020

@rniwa could you please clarify your objection from this comment?

I believe you are objecting to any deprecation or change in behavior for inputmode=none.
This proposal doesn't suggest we deprecate inputmode=none.

I think the root of the confusion is that inputmode=none has some overlap with the VK APIs (but isn't a replacement) so at one point it was suggested that we could deprecate inputmode=none since we would have the more capable VK API. @rniwa is saying that wouldn't be acceptable as inputmode=none has been promoted and is being used to dismiss the keyboard in some scenarios.

We therefore abandoned the idea of deprecating inputmode=none (in response to @rniwa's comment), but that doesn't mean we don't need the VK API. The VirtualKeyboard interface and the virtualKeyboardPolicy attribute provides authors with the ability to:

  1. Request that the keyboard be hidden
  2. Request that the keyboard be shown
  3. Handle an event for when the keyboard changes its visibility and adjust to the size and position of the keyboard
  4. Prevent the visual viewport from being resized due to the appearance of the VK

inputmode=none only addresses the first of those capabilities and comes with side effects that are discussed in the same github issue (see here).

Hope that helps clarify things. I don't believe there is any conflict here.

@rniwa
Copy link

@rniwa rniwa commented May 21, 2020

I've been on a medical leave. Hopefully @whsieh can clarify our position here.

@cynthia
Copy link
Member

@cynthia cynthia commented May 26, 2020

Given the comments from @BoCupp-Microsoft above, we'd like to hear if there is still strong objection with regards to the new proposal, as we don't find it as controversial as we initially thought. Can @whsieh or someone else comment on the objection points?

@whsieh
Copy link

@whsieh whsieh commented Jun 12, 2020

Sorry for the delay; I have a couple of questions below:

For (3) — it wasn’t totally clear to me why the visual viewport API isn’t already sufficient for this purpose. Is it because the VisualViewport API isn’t compatible with devices with multiple screens, when the software keyboard only takes up space in one of the screens?

For (4) — I would like a bit more clarity on what it means to prevent the visual viewport from being resized due to the appearance of the virtual keyboard. Does that mean the visual viewport should be “pushed upwards” (and retain its size) when the virtual keyboard is brought up? Or that the visual viewport should overlap with the bounds of the software keyboard?

@BoCupp-Microsoft
Copy link

@BoCupp-Microsoft BoCupp-Microsoft commented Jun 13, 2020

For (4) — I would like a bit more clarity on what it means to prevent the visual viewport from being resized due to the appearance of the virtual keyboard. Does that mean the visual viewport should be “pushed upwards” (and retain its size) when the virtual keyboard is brought up? Or that the visual viewport should overlap with the bounds of the software keyboard?

The latter. The visual viewport will overlap with the bounds of the virtual keyboard when the author sets navigator.virtualKeyboard.overlaysContent = true.

For (3) — it wasn’t totally clear to me why the visual viewport API isn’t already sufficient for this purpose. Is it because the VisualViewport API isn’t compatible with devices with multiple screens, when the software keyboard only takes up space in one of the screens?

Philosophically the visual viewport API isn't sufficient because when then the keyboard overlays the content the visual viewport remains unchanged. IMO the visual viewport API should not describe events that don't affect the size of the visual viewport.

Additionally, as you mention, the visual viewport describes a rectangular viewport, which is problematic for virtual keyboards on dual screen devices because those would create an L-shaped visual viewport if we don't want to waste space as shown in this explainer.

@plinss plinss removed this from the 2020-04-27-week milestone Jun 19, 2020
@plinss plinss added this to the 2020-06-22-week milestone Jun 19, 2020
@plinss plinss removed this from the 2020-06-22-week milestone Jun 24, 2020
@BoCupp-Microsoft
Copy link

@BoCupp-Microsoft BoCupp-Microsoft commented Sep 29, 2020

@rniwa following up on one of your earlier points which I didn't respond to yet:

On iPad, we have split keyboard and floating keyboard both of which are keyboards that simply overlays on top of the app / web content. Presumably in these cases, keyboard-inset-height would be 0.

Yes the height would be 0. We wrote something about floating keyboard UI in the explainer under the non-goals section. In short, we don't want the author to compete with the user when the user is in charge of positioning the keyboard.

@BoCupp-Microsoft
Copy link

@BoCupp-Microsoft BoCupp-Microsoft commented Sep 29, 2020

Do we want to have a virtual F2F discussion on this topic? I'm willing to meet up just with the people on this thread or at a larger venue during virtual TPAC. Let me know if anyone has suggestions for a venue.

Also, I think we're trending towards agreement. I'm interested in finding collaborators that would like to co-author a spec. Please let me know if your interested. Past experience writing specs would be a big plus. :-)

@plinss plinss removed this from the 2020-09-21-F2F-Cork milestone Oct 12, 2020
@plinss plinss added this to the 2020-10-12-week milestone Oct 12, 2020
@rniwa
Copy link

@rniwa rniwa commented Oct 13, 2020

It seems worth discussing this as a breakout session: https://www.w3.org/wiki/TPAC/2020/SessionIdeas

@torgo
Copy link
Member

@torgo torgo commented Oct 13, 2020

Sounds like this discussion is going to happen at TPAC. That's great. Please let us know if you want someone from the TAG to join. I'll bump this issue until after TPAC and we can pick it back up then.

@torgo torgo removed this from the 2020-10-12-week milestone Oct 13, 2020
@torgo torgo added this to the 2020-11-02-week milestone Oct 13, 2020
@plinss plinss removed this from the 2020-11-02-week milestone Oct 14, 2020
@plinss plinss added this to the 2020-11-09-week milestone Oct 14, 2020
@BoCupp-Microsoft
Copy link

@BoCupp-Microsoft BoCupp-Microsoft commented Oct 17, 2020

@rniwa @torgo @kenchris @snianu @cynthia @whsieh @bokand I've proposed a breakout session here called Virtual Keyboard Control. Right now I have the session marked so that it will be scheduled during the 2pm UTC golden hour that was suggested on either Wednesday, Thursday or Friday of the breakout week. Hopefully that will work for anyone that would like to attend. If not, let me know and I can always just schedule something outside of TPAC.

blueboxd pushed a commit to blueboxd/chromium-legacy that referenced this issue Oct 31, 2020
This patch contains the geometrychange JS event and overlayscontent
implementation of the new VirtualKeyboard APIs on Android.
This is a new feature that would let web authors to control whether
they want the default occlusion of web content and changing of visual
/layout viewport in response to VK visibility or not. The explainer
has some examples as to where this would be useful and how it would
be used by the web authors.

Test page: https://bocupp-microsoft.github.io/VirtualKeyboard/geometrychange.html

Explainer:
https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/VirtualKeyboardAPI/explainer.md

Design Doc: https://docs.google.com/document/d/1I0LUNxK_gP5IaNQsbYN6gL6Zpm71XzduCKkublU5o5Q/edit?usp=sharing
Check the Android Design section for more info on the proposed design

We also discussed in the TAG review about this event and haven't seen
any objections yet. w3ctag/design-reviews#498

Bug: 856269

Change-Id: I42a7fa737dec06be04b8b1a59b3f511d3678c791
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2463811
Reviewed-by: Avi Drissman <avi@chromium.org>
Reviewed-by: Bo <boliu@chromium.org>
Reviewed-by: Theresa  <twellington@chromium.org>
Reviewed-by: Friedrich [CET] <fhorschig@chromium.org>
Commit-Queue: Anupam Snigdha <snianu@microsoft.com>
Cr-Commit-Position: refs/heads/master@{#822920}
@kenchris
Copy link

@kenchris kenchris commented Nov 10, 2020

@BoCupp-Microsoft Can you post a summary of the relevant discussion at TPAC?

@torgo torgo removed this from the 2020-11-09-week milestone Nov 10, 2020
@torgo torgo added this to the 2020-11-16-week milestone Nov 10, 2020
@snianu
Copy link
Author

@snianu snianu commented Nov 13, 2020

Meeting minutes: https://www.w3.org/2020/10/30-editing-minutes.html
Summary of the discussion:
We discussed about the VirtualKeyboard APIs and how it affects UX on both dual and single screen devices. The presentation was mainly focused on describing the problems faced by web authors while dealing with virtual keyboard(VK) on touch devices where the VK is the primary source of text input for users. We showed a demo of the VK APIs and how the authors could use it to provide a much better UX experience on their sites.
Demo page: https://bocupp-microsoft.github.io/VirtualKeyboard/geometrychange.html
Slides: https://cuppfamily-my.sharepoint.com/:p:/g/personal/bo_seattlecupps_com/EVoI7Hp5An1HokezOTE3hK0BLMT2UsL1k7-ryO8-8efv6w?e=EplB9S

Issue discussions:

  • Concerns about VK APIs being abused and sites behaving poorly because of that:
    We currently have sticky user activation and active document as a requirement to restrict showing the VK. VK APIs are also an opt-in mechanism where authors have to explicitly indicate that they would like to control the VK behavior. Currently the show/hide behavior can be abused by using inputMode property.
  • Is navigator the right place?
    We didn't get a chance to discuss this in detail, but I haven't seen any objections in this issue after the latest response.
  • Relationship with the VisualViewport API and concerns in general regarding creating a separate API to handle overlaying of content in response to VK show/hide:
    We had extensive discussions on this issue. We generally agree that having a separate event describing the overlay property and geometry of the VK helps the authors make better decision as to how to position specific parts of the webpage in response to VK occluding the content of the webpage.
  • Have VK show/hide APIs return a promise
    We all agreed that returning a promise when authors call show/hide helps to test the APIs and also make decisions when the calls failed to show/hide the VK. This needs more discussion as we are not entirely sure that returning an error code helps the authors in a meaningful way.
  • We are also looking for collaborators to write a spec for these APIs.
    @BoCupp-Microsoft Please let me know if I missed anything.

@kenchris
Copy link

@kenchris kenchris commented Nov 17, 2020

I also saw that you added support for using CSS environment variables for the geometry in addition to the JS API 👍

@cynthia
Copy link
Member

@cynthia cynthia commented Nov 18, 2020

Based on the discussions above - we took this up in the plenary call this week and decided we're happy to see this move forward. Thanks for bringing this to our attention!

@BoCupp-Microsoft
Copy link

@BoCupp-Microsoft BoCupp-Microsoft commented Apr 23, 2021

Hello TAG,

Based on the comment from @cynthia here, could you update the labels on this issue to include Resolution: satisfied?

Thanks for your help on this!
Bo

Tagging a couple other members so this comment gets seen: @atanassov @kenchris

@cynthia
Copy link
Member

@cynthia cynthia commented Apr 24, 2021

Done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests