How does cE=typing support IME text replacement?

[BenjamP]

IMEs do inline composition and can even replace existing text (using conversion for instance). This means they expect to be able to replace existing content. Since we don't support replace/delete, how does this work?

IMEs also have special styling for compositions. Do we have special considerations there?

[teleclimber]

Also related is the "overtype" situation: #28

If I remember right we went from cE=cursor to cE=typing because we wanted IMEs and autocorrect and other hard-to-implement things to be done by the UA or OS. I think this is still the case. So I think we might explore the possibility of giving cE=typing the ability to alter/delete the contents of text nodes to make it handle autocorrect and IMEs and overtype.

[BenjamP]

One suggestion we had in the last Editing phone conference (on Jan 9, 2015) was to disable inline composition for V1. Then IMEs would automatically fallback to composition in the candidate window.

[BenjamP]

After reviewing the fallback plan with experts at Microsoft, we believe that this is not a good approach, even for V1. Users expect inline composition to work, and the fallback experience is not as good as the inline experience. Any serious consumer of contentEditable will expect inline composition.

[kojiishi]

Thank you Ben for checking this, understood, we need to spend time on this one.
One big first question for me then is that how much we want to make this uncommitted states visible to editor developers.
I'm leaning to make this as invisible as possible for benefits like:

• Apps designed for non-IME require less (or ideally zero) modifications for IME
• We don't have to worry about special styling
• We also don't have to worry about overtype/replace/delete because all happens within the invisible

But not entirely sure at this point if we can make a flawless design in this direction, and the last point may need to be addressed anyway for AutoCorrect or other features we consider required for v1.
Thoughts?

[rniwa]

Robin suggested an approach that might be workable, which is to insert an placeholder element and insert uncommitted text into its user agent "shadow DOM".

[teleclimber]

in #39 @kojiishi brings up the issue where some manipulations by editor dev can cause uncommitted IME text to be committed. This is something we should take into account when figuring out how to do IMEs.

[teleclimber]

I'm worried that inserting a placeholder element doesn't fit with cE=typing's "text-only" philosophy. Inserting an element means the UA will be breaking a text nodes in two, which is exactly the kind of thing we were looking to avoid.

[rniwa]

If we're going with a place holder element route, we could make author scripts place the placeholder at the right place instead of UA automatically inserting it.

[kojiishi]

The placeholder route can be further adjusted to make the text node as a shadow host. Currently Blink does not support creating shadow roots against text node, but this could be extended, or we could create similar mechanism without directly using shadow DOM as is.

So the idea is that, we don't have to say whether it's shadow or not, but when IME inline starts, un-committed strings do not appear in the text node. It will be displayed, and intrinsic size of the text node changes, but DOM does not change. The text appears in the current text node only when committed.

Does this route look good?

[kojiishi]

A few things we might want to consider if we're going to this route:

• Make sure editor developers are happy with this route. Any cases they want to be more aware of IME states and uncommitted strings?
• Ensure this is reasonable way to implement.
• Define when browsers would commit/cancel uncommitted strings (or make sure not defining this won't trouble interoperability and editor developers).
• Anything else?

[teleclimber]

If we're going with a place holder element route, we could make author scripts place the placeholder at the right place instead of UA automatically inserting it.

That goes against Koji's idea that IMEs should work without developer involvement. IMEs would only work if devs take the time to add that logic to their editor's code. And it sounds like it's not going to be trivial.

Personally I'm not thrilled about using custom elements as a requirement to using cE=typing. Custom Elements is a new technology, and judging by the number of emails that come through my inbox it sounds like there is a lot of work still going on. It seems that if cE=typing becomes dependent on custom elements we inherit a whole lot of uncertainty.

Can we backtrack a bit? contenteditable=true has support for IMEs baked in, with no involvement from the editor devs. If we allow cE=typing to alter the contents of text nodes when an IME is used, wouldn't that be 80% of the solution? cE=typing can already alter contents of a text node during traditional keyboard typing. Is this really that different?

What are the issues that are causing us to want to reinvent the way we handle IMEs?

[kojiishi]

Personally I'm not thrilled about using custom elements as a requirement to using cE=typing. Custom Elements is a new technology, and judging by the number of emails that come through my inbox it sounds like there is a lot of work still going on.

Custom elements and shadow DOM are two different animals, and Shadow DOM is much more stable. Chrome has been using this "shadow DOM" to implement input and some other elements internally, but that's not defined in the spec. It's an implementation details.

What are the issues that are causing us to want to reinvent the way we handle IMEs?

It's not really a reinvent. Let me re-phrase the question: should IME interim states (un-commited strings) visible to DOM or not? Existing implementations vary: Windows does not expose them unless apps opt-in, I don't know how OS X/Linux API does for this. Browsers used to expose by default, though I'm not sure if this was spec'ed and interoperable. Probably not.

By exposing, we'll need to define replace/delete behavior, since IME makes a lots of inserts/deletes/replaces during the interim states.

If we change it not to expose, all what IME does is inserting text on commits, which aligns well with cE=typing, and we can resolve this issue. Since what we're trying to do is to minimize browsers doing special operations to the DOM, I think not exposing aligns to our overall direction.

[rniwa]

Like I stated elsewhere, I find the behaviors IME and spellchecking need to be quite similar. Both features require replacing a part of text and show special markings (e.g. waved underlines) under some text.

I'd say both of these features as well as bidirectional text MUST be fully supported in v1.

[BenjamP]

We need to agree on a major point before we can move forward on this: is it okay to require sites to do special work to support a few key scenarios involving text replacement- certainly IMEs, maybe spellchecking, maybe overtype (insert key).

Sites already know that they have to do special work to support basic editing. For that reason, I believe the target audience for ce=typing should be sophisticated developers writing frameworks like ckeditor and fidus writer. Adding a requirement for special handling of spellchecking and IMEs does not seem like too much work for those folks. It makes the architecture of ce=typing much cleaner as well. So I say we require this special handling (you must convert the affected range to a single text node in order for overtype/IME/spellcheck to work).

In the end this doesn't make site developer's lives harder, because they should be using frameworks anyway and this will be invisible to them.

[rniwa]

I think we should enforce this requirement in all cases for consistency so that basic text input doesn't work without it.

[BenjamP]

I agree with Ryosuke. I'm happy to write up a proposal unless anyone thinks the points we each made above are fundamentally wrong?

[rniwa]

A concrete proposal on public-webapps would be great!

[BenjamP]

Why not the Editing TF?

[rniwa]

That works too but it might be good to get a broader perspective/opinion on the matter.

[BenjamP]

Let's talk about this in our group and once we agree bring it out broader. I'm going to update our documents with some new ideas based on conversations in these bugs and see what you all think.

[teleclimber]

you must convert the affected range to a single text node in order for overtype/IME/spellcheck to work

I thought we established that spellcheck/autocorrect had to work across style tags (see #37). So how how can affected contents be placed in a single text node? Or am I missing something?

[BenjamP]

That conversation appears to have concluded that they don't need to work across styles:

Me:
I agree with that Ryosuke. A replace event could require that the site ensure that the selection is completely contained inside a single text node in order to have a default action. It could then handle IME, spellchecking, and maybe overtype.

Ryosuke:
That's a good point! We could even throw some error if the selection is anchored across non-text nodes inside the default event action.

[rniwa]

Hm... I think we do need to support spellchecking that crosses element boundary since certain spellchecking and grammar correction features work at paragraph levels. It would be unreasonable to expect authors to strip all styling before performing spellchecking.

[BenjamP]

I think we should allow scanning of the document at any level, but replacement should only be allowed in a single text node.

[rniwa]

Why can't we send a "replacement" intent with a range?

[BenjamP]

We can for spellcheck/autocorrect if we want. But we can't really do that for IME, so I thought we wanted to unify the way IME/spellcheck was handled for simplicity.

[teleclimber]

Question: Do IMEs have to work across style boundaries? Do they work across style boundaries in current versions of UAs?

[rniwa]

IME does not work across style boundaries but I could imagine some platform may want to support something like that.

[teleclimber]

Here is a breakdown the characteristics of each feature as I understand them:

Overtype: intent is to replace a range with new content. Range coincides with Selection.

Spellcheck/Autocorrect: intent is to replace an arbitrary range with new text. Also, some ranges may get decorated by the UA (red squiggles).

IME: replaces arbitrary range with new content. Some ranges may get decorated by UA. UA may also create interactive dialogs/pickers and may capture the focus to allow interaction with these. UA may want to temporarily replace a range with new content when user hovers over a choice.

(I'm not very familiar with IMEs. Are there other things they do we should be considering here?)

---

We can see that content replacement is a common element to all. I'd be in favor of using a common "replace" intent that includes a range and the new text. That intent would probably indicate whether the intent came from overtype or autocorrect or something else, but the editor dev will treat it all the same. The Range could cross style boundaries.

Text decoration by the UA is also present. In my view the editor dev doesn't really need to know about that, as long as the UA is able to do the decoration internally without affecting the dev-facing DOM. This is currently the way things work in cE=true (I think?) and it's fine IMO.

IME brings some special behaviors to the table. I feel like the biggest gotcha with IMEs is that if the editor dev does something unexpected (like resetting the Selection, as was noted elsewhere) the IME will commit automatically. We might consider adding some sort of signaling (events or maybe attributes on Selection) to help editor devs avoid breaking IMEs.

The other trick about IMEs is that they like to replace text in the editor on a temporary basis (mouseover a choice). We might consider adding a "temporary" flag to the replace intent. This would instruct the dev editor to execute the replacement in the View, but that it is not meant to be committed to disk or sent over the network or whatever.

So I think that if we have a Replace Intent with "temporary" flag, and we allow the UA to decorate editable text, and we add some signaling on the state of IME input, we're most of the way there.

[johanneswilm]

After reading this (very long) conversation, I must say that indeed, it's not a problem to handle IMEs and spellcheck/auto correct intents differently for complex editor libraries, but that it makes things much easier if one is just dealing with a single text node. As far as I can tell, noone has come up with usecases for IME input that goes across elements, so let's not overcomplicate things. Let's stick with the simple version for now.

[johanneswilm]

@kojiishi Could you check whether this works for your simplified IME input idea? Specifically, have a look at https://github.com/w3c/editing-explainer/blob/gh-pages/contentEditableTyping.html#L261-L279