VirtualKeyboardAPI spec draft #309

gked · 2021-06-22T17:28:33Z

This is the first draft of the spec to iterate on.
No tests, so far, just the wording around the behavior of the API.
There are a few respec errors that we'll be looking at as well.

The following tasks have been completed:

Confirmed there are no ReSpec/BikeShed errors or warnings.
Modified Web platform tests (link to pull request)

Implementation commitment:

WebKit ()
Chromium (https://bugs.chromium.org/p/chromium/issues/detail?id=856269)
Gecko ()

yoavweiss · 2021-06-25T12:49:13Z

docs/virtualkeyboard/index.html

+                  <li>If the <a>document</a> doesn't have <a href="https://html.spec.whatwg.org/multipage/interaction.html#sticky-activation">sticky user activation</a>, abort these steps.
+
+                  </li>
+                  <li>Trigger {{VirtualKeyboard/show()}} call that invokes a system API to show the VK.


That seems kinda circular. It'd be good to define what the call actually does.

For example, I'm guessing it's impacting boundingRect in some way.

Similarly, it probably fires the geometrychange event at some point.

It might be best to define the processing of what happens when the virtual keyboard is displayed and folded and then hook into that processing from these explicit methods, but also from places where the UA can decide to do that automatically (e.g. from editable content when it's getting focus)

yoavweiss · 2021-06-25T12:49:50Z

docs/virtualkeyboard/index.html

+                  The method must follow these steps:
+                </p>
+                <ol>
+                  <li>If the <a>document</a> doesn't have <a href="https://html.spec.whatwg.org/multipage/interaction.html#sticky-activation">sticky user activation</a>, abort these steps.


If I'm reading the sticky activation algorithm correctly, it applies to a Window object, not a Document.

It would be good to clarify which Document (or Window) we're talking about

yoavweiss · 2021-06-25T12:55:46Z

docs/virtualkeyboard/index.html

+                <ol>
+                  <li>If the <a>document</a> doesn't have <a href="https://html.spec.whatwg.org/multipage/interaction.html#sticky-activation">sticky user activation</a>, abort these steps.
+                  </li>
+                  <li>Trigger {{VirtualKeyboard/hide()}}call that invokes a system API to hide the keyboard.


Same comments as above

yoavweiss · 2021-06-25T12:57:12Z

docs/virtualkeyboard/index.html

+                Platform heuristics may impose additional restrictions on VK {{VirtualKeyboard/show()}} and {{VirtualKeyboard/hide()}}.
+              </p>
+              <p class="note">
+                A few people have expressed concerns around {{VirtualKeyboard/show()}} and {{VirtualKeyboard/hide()}} methods not being promise-based. However, we couldn't find any real use cases around needing promises in this type of operation.


Examples below are showing them returning a Promise. Would be good to clarify which variant is implemented.

yoavweiss · 2021-06-25T13:00:08Z

docs/virtualkeyboard/index.html

+            </dt>
+            <dd>
+              <p>
+                The attribute enables the author to opt-out of resizing the visual viewport.


Would be good to define a setter that impacts the processing and does the actual opt-out.

yoavweiss · 2021-06-25T13:00:33Z

docs/virtualkeyboard/index.html

+            </dt>
+            <dd>
+            <p>
+                The attribute reports the intersection of the VK with the layout viewport in client coordinates.


You should define a getter that defines what's returned here

yoavweiss · 2021-06-25T13:04:47Z

docs/virtualkeyboard/index.html

+        </pre>
+        <p>The <dfn><code>virtualKeyboardPolicy</code></dfn> is an <a href=https://html.spec.whatwg.org/multipage/common-microsyntaxes.html#enumerated-attribute>enumerated attribute</a> whose keywords are the empty string, auto, and manual.
+        <p>When specified on an element that is a contenteditable host, <dfn><code>auto</code></dfn> causes the corresponding editable element to automatically show the VK when it is focused or tapped & <dfn><code>manual</code></dfn> 
+            decouples focus and tap on the editable element from changes in the VK’s current state - the VK remains as it was.


Similarly to the above, you probably want to define the processing for this attribute: what happens when it's set, changed, removed, etc.

yoavweiss · 2021-06-25T13:05:44Z

docs/virtualkeyboard/index.html

+                    <th>Value
+                <tr>
+                    <td><dfn>keyboard-inset-top</dfn>
+                    <td><a href="https://drafts.csswg.org/css-values-3/#length-valuehttps://drafts.csswg.org/css-values-3/#length-value"> length</a>


You probably want to define the semantics of each of these variables and what its value is set to.

gked · 2021-06-28T19:24:04Z

@yoavweiss We addressed your feedback. Could you please let us know if they resolve your questions? One thing, I am trying to resolve currently is making adding the ability to make a PR readable with .pr-preview.json file.

BoCupp-Microsoft · 2021-06-30T00:52:27Z

docs/virtualkeyboard/index.html

+<body>
+    <section id='abstract'>
+        <p>
+            The {{VirtualKeyboard}} API provides web developers a more granular control of on-screen keyboard (virtual keyboard or VK). The API also provides information about VK geometry, such as rect occlusion and gives ability to opt-out from resizing of the visual viewport in the browser.


nit: grammar issue with "a more granular control"

Suggested new abstract: The {{VirtualKeyboard}} API provides authors with greater control over the visibility of the virtual keyboard (VK), and greater ability to adapt the layout of web pages when VK visibility changes.

BoCupp-Microsoft · 2021-06-30T00:55:49Z

docs/virtualkeyboard/index.html

+        <h2>Introduction</h2>
+        <P>This section is non-normative.</P>
+        <p>
+            The Virtual Keyboard (VK) is the on-screen keyboard used for input in scenarios where a hardware keyboard may not be available. Unlike hardware keyboard, VK can adapt its shape to optimize for the expected type of input.


Suggest cutting this sentence as it doesn't seem relevant to the API: Unlike hardware keyboard, VK can adapt its shape to optimize for the expected type of input.

I think it was useful in the explainer since we were pointing out a gap that inputMode was overloaded to control both shape and visibility and couldn't do both at once, but in the spec I don't see that it's necessary.

agreed - updated.

BoCupp-Microsoft · 2021-06-30T01:06:13Z

docs/virtualkeyboard/index.html

+        <P>This section is non-normative.</P>
+        <p>
+            The Virtual Keyboard (VK) is the on-screen keyboard used for input in scenarios where a hardware keyboard may not be available. Unlike hardware keyboard, VK can adapt its shape to optimize for the expected type of input.
+            In addition, user agents respond to the presence of the VK, without any exposure of this information to web developers. This document proposes a new web API surface for VK that developers can use to enable better customization of their webpage's content and experiences.


I think it would be useful in this paragraph to expand on how user agents currently respond to the presence of the VK and what alternatives this API offers. As is I don't think it adds anything compared to the abstract.

Maybe: User agents respond to the presence of the VK by:

Repositioning the user agent above the VK

Reducing the size of the layout viewport so the VK doesn't occlude it

Reducing the size of the visual viewport and padding the layout viewport to ensure it can be shifted above the VK

This API provides a fourth option that allows the user agent to leave its layout and visual viewports unchanged and instead provide information about the intersection of the VK and layout viewport so that the author can adapt the layout of their web pages using JavaScript or CSS environment variables.

It would be good to have graphics illustrating the difference between these behaviors.

BoCupp-Microsoft · 2021-06-30T01:08:13Z

docs/virtualkeyboard/index.html

+
+            [Exposed=Window]
+            interface VirtualKeyboard : EventTarget {
+                undefined show();


I think this should be void instead of undefined.

https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/virtualkeyboard/virtual_keyboard.idl;l=1?q=virtual_keyboard.idl&sq=&ss=chromium

we were getting respec errors when we used void. Web IDL recommends using undefined instead of void https://heycam.github.io/webidl/#idl-undefined

BoCupp-Microsoft · 2021-06-30T01:12:43Z

docs/virtualkeyboard/index.html

+                    <li>If the browsing context's <a href="https://html.spec.whatwg.org/multipage/browsers.html#active-window">active window</a> does not have <a href="https://html.spec.whatwg.org/multipage/interaction.html#sticky-activation">sticky user activation</a>, abort these steps.
+                  </li>
+                  <li>
+                    If the focused element's <a href="https://html.spec.whatwg.org/multipage/interaction.html#attr-contenteditable">contenteditable</a> value is false, then abort these steps.


I don't think this is meant to be limited to contenteditable. It should say something like: If the focused (is focused the right word) element is not editable...

We want to include textareas, text-like inputs, and any inputs into which the user can type (in the near future EditContext associated elements too).

BoCupp-Microsoft · 2021-06-30T01:21:52Z

docs/virtualkeyboard/index.html

+               </dd>
+               <p class="note">
+                Platform heuristics may impose additional restrictions on VK {{VirtualKeyboard/show()}} and {{VirtualKeyboard/hide()}}. e.g., On Windows the pointer type has to be
+                touch or pen.


nit: other paragraphs don't have hard line breaks dividing their content... remove this one?

BoCupp-Microsoft · 2021-06-30T01:22:10Z

docs/virtualkeyboard/index.html

+                touch or pen.
+              </p>
+              <p class="note">
+                A few people have expressed concerns around {{VirtualKeyboard/show()}} and {{VirtualKeyboard/hide()}} methods not being promise-based. However, we could not find any real use cases around needing promises in this type of operation.


Can you link to an issue?

BoCupp-Microsoft · 2021-06-30T01:28:27Z

docs/virtualkeyboard/index.html

+            </dt>
+            <dd>
+              <p>
+                To set the attribute to opt-out of resizing the visual viewport, 


Can you say something slightly stronger here: a user agent MUST NOT resize its layout viewport or visual viewport when overlaysContent is true?

Also, I think only top-level browsing contexts are allowed to set this, right? This would probably be the place to mention requirements like that. I think there are other requirements too (secure origin maybe)?

Updated the spec to reflect on your first comment.

Is defining SecureContext in web idl enough so we don't have to mention it here?

BoCupp-Microsoft · 2021-06-30T01:39:33Z

docs/virtualkeyboard/index.html

+            </section>
+    </section>
+    <section>
+        <h2>Privacy and Security Considerations</h2>


Several pieces of information in this section seem like a set of claims that this API doesn't come with significant risk to privacy or security. I don't think we need to cover that. I expected this section to enumerate important implementation details that UAs may or must follow to ensure a secure and privacy preserving implementation. Additionally, I think it would be appropriate to enumerate the responsibilities of authors, if any, to ensure they don't create privacy or security issues when using the API.

Maybe IndexedDB is a good example.

Updated. Is this what you had in mind?

BoCupp-Microsoft · 2021-06-30T01:50:20Z

docs/virtualkeyboard/index.html

+        </section>
+    </section>
+    <section>
+        <h2>Examples</h2>


I think there were good images in the explainer to go with these examples. Can you include them to help illustrate the impact of the code?

johanneswilm · 2021-07-01T23:43:18Z

.pr-preview.json

@@ -1,4 +1,4 @@
 {
-    "src_file": "index.html",
+    "src_file": "\\docs\\virtualkeyboard\\index.html",


are those Windows-slashes?

Thanks! fixed. This is probably why the preview functionality didn't work :)

BoCupp-Microsoft

This looks good enough to me to merge and subsequent issues we can deal with in smaller easier to review PRs.

Grisha Lyukshin and others added 11 commits March 2, 2021 13:53

initial vkAPI spec commit

c6d60f3

added privacy section and additional references

7d83f4c

Added CS env variables.

816e1a2

added new sections and simplified the flow a little bit

f28e581

Added links to sticky user activation.

258ab19

initial vkAPI spec commit

b0686d8

added privacy section and additional references

a720a02

Added CS env variables.

6800df5

added new sections and simplified the flow a little bit

461dade

merge fix

dd7c3b7

partial changes to spec wording

78cfebc

gked requested review from johanneswilm, BoCupp-Microsoft and snianu June 22, 2021 17:28

yoavweiss reviewed Jun 25, 2021

View reviewed changes

Grisha Lyukshin added 4 commits June 25, 2021 17:59

addressing PR feedback

4f4c985

removing ce which was used for testing

cf0f3ce

removed redundant .pr-preview.json file

b30e003

updating respec url to defer for PR preview

458857a

BoCupp-Microsoft reviewed Jun 30, 2021

View reviewed changes

Grisha Lyukshin added 2 commits June 30, 2021 18:37

addressing partial changes

d621cce

addressing the 2 of the last 3 comments of this iterations

95106b4

johanneswilm reviewed Jul 1, 2021

View reviewed changes

Grisha Lyukshin added 3 commits July 2, 2021 10:52

fixing Windows to Unix path variant

3f4a6d3

addressing sec and priv feedback

3057af0

addressing more feedback

ab9bec0

BoCupp-Microsoft approved these changes Jul 6, 2021

View reviewed changes

gked merged commit 73389dc into w3c:gh-pages Jul 6, 2021

VirtualKeyboardAPI spec draft #309

VirtualKeyboardAPI spec draft #309

Conversation

gked commented Jun 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gked commented Jun 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gked Jul 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BoCupp-Microsoft left a comment

Choose a reason for hiding this comment

gked Jul 1, 2021 •

edited

Loading