Adding support for mobile

[matthew-carroll]

Super Editor is committed to delivering document rendering and editing on all platforms, including iOS and Android. This proposal outlines some of the challenges associated with mobile support and how Super Editor might overcome those challenges.

Why Super Editor doesn't support mobile

The primary issue is mobile text input. All text input on iOS and Android devices begins at the keyboard, flows to the IME service, is dispatched to the active application, is piped from the mobile SDK side over to the Flutter/Dart side, and is exposed in Flutter via TextInput. To handle text input requests from TextInput, one must register as a TextInputClient. Only a single TextInputClient can register to receive information from TextInput at any given moment.

The current implementation of Super Editor does not use TextInput at all. Instead, Super Editor listens for every key press on the user's keyboard with a RawKeyboardListener and then maps those key presses to an associated editor action. This approach works reasonably well in desktop/web environments. However, RawKeyboardListener does not function at all on mobile. Despite the fact that the user taps on something that looks like a keyboard on their phone, it isn't really a keyboard, and it doesn't expose individual keypresses as one might expect. Mobile devices intercept keys that the user presses on the keyboard and, possibly, alters the content. Here are some examples to consider:

• you type "huose" and your phone keyboard recommends changing the entire word to "house"
• you type "thea" and your phone offers to complete the word to "theater"
• you type ":" and your phone offers a collection of emojis to choose from

On mobile, text entry requires a back-and-forth conversation between the app and the operating system, because your keyboard is engaged in a back-and-forth conversation with you. TextInput and TextInputClient are the interfaces that Flutter exposes to engage in this 2-way conversation.

Challenges to fulfill the TextInputClient API

The greatest challenge, and risk with utilizing the TextInputClient API is that we give up a lot of control.

Narrow and Specialized APIs

All things considered, the TextInput/TextInputConnection/TextInputClient APIs are relatively narrow, and some of the APIs are quite specialized. Here is an example of a very specialized API:

Sends the coordinates of caret rect. This is used on macOS for positioning the accent selection menu.

void setCaretRect(Rect rect)

https://api.flutter.dev/flutter/services/TextInputConnection/setCaretRect.html

Will all TextInput requirements make sense for Super Editor, and will Super Editor be able to accomplish all desired edits when using the available TextInput APIs?

Cross-Boundary Edits

The TextInput APIs were designed in a world with independent text fields. In Super Editor, the user might select across part of a paragraph, and across multiple list items, at the same time.

Will TextInput APIs work as expected when a selection spans multiple text content blocks?

Multi-Media Editing

Unlike a typical mobile text field, Super Editor displays images, horizontal rules, and any other desired content within a single document.

Will TextInput APIs get in the way of Super Editor's multimedia editing capabilities?

Existing Work

@osaxma has experimented with replacing the standard Super Editor interaction system with TextInputClient.

Here is a demo:

super_editor_mobile_recording_01.mp4

The initial results look promising.

WORK IN PROGRESS

[osaxma]

Hi @matthew-carroll

This comment will primarily focus on the Document Editing using the soft keyboard without getting into the specialized APIs. I'll write a separate comment later about the rest. Please let me know if anything isn't clear.

Soft Keyboard Editing

Objectives:

• Support editing non-textual nodes using the soft keyboard (e.g. delete a selected image using backspace)
• Support native Text Editing features including auto-correct, suggestions, swipe to text and voice-dictation.

Background:

Before diving into the details, it's important to mention the following:

Using TextInput:

• In Flutter, EditableTextState is a TextInputClient and it maintains a TextInputConnection.
• The text editing communication between the client and system is done by sending TextEditingValue as a message which contains the plain text, selection, and composing region.
• During the lifetime of the connection, there exist two TextEditingValues which are:
  • Remote TextEditingValue which represent the System's Input Control state.
  • Local TextEditingValue which represents the client state.
• It's the EditableTextState responsibility to keep the values in sync.
• The system sends changes to the EditableTextState through TextInputClient.updateEditingValue (i.e. EditableTextState.updateEditingValue) so the local value can stay in sync.
• EditableTextState sends changes to the system using its connection (i.e. TextInputConnection.setEditingState) to update or reset the remote value.

Similarly, DocumentInteractorState can be implemented as a TextInputClient where it maintains its own TextInputConnection.

Detecting Backspace:
(flutter/issues/14809) Detect when delete is typed into a TextField

This issue originates from TextInput. Simply, TextInput, or the system, does not send a delete event when the key press doesn't cause a change to the text. This includes an empty text or when the selection is collapsed at the beginning of the text (i.e. offset 0). Based on the discussion in the linked issue, one approach is to place a special invisible character (i.e zero-width space character or zwsp for short) at the beginning of the text to detect when a deletion occurs.

In this use case, the zwsp can be used in the following manner:

• The value is only added to the remote TextEditingValue at offset 0.
• The value will not be added anywhere to the document being edited (i.e. MutableDocument). In other words, even if a visible value was used, it'll not appear in the rendered document since it doesn't exist there.
• This means whenever DocumentInteractorState.updateEditingValue receives an event from the system, the zwsp character must be dealt with to detect deletion/insertion or adjust the selection.

Dealing with Newlines:

• newlines can be detected in two ways:
  • Listening to actions at TextInputClient.performAction.
  • If the TextInputConfiguration.inputMethod == TextInputAction.newline, the newline carriage (i.e. \n) will appear in the remote TextEditingValue in addition to a newline action event from TextInputClient.performAction.
  • Using TextInputAction.newline is necessary to show the newline symbol on the software keyboard.
• all newlines are treated as hard-return since the soft-return (shift + enter) concept doesn't exist in mobile.
  • [alternative option] treat \n as a soft-return and \n\n as a hard return at least for SuperEditor.

Overview:

Despite the limitation of having a single connection to TextInput, the remote value can be updated to represent what's being edited in the document based on the selected node(s). This entails updating the remote value, if necessary, after each selection change. On the other hand, this limits the system's control over the document since it can only edit whatever it was given.

Note: it's impractical to set the entire document as a remote TextEditingValue. First of all, the TextEditingValue only accepts plain text. Even if the document is serialized as text, selection info is lost due to the difference in the text length. Detecting changes would be another complexity (e.g. image serialized as text and part of the text is deleted). In addition, computing all of the above for every keyboard press could present an issue, especially for larger documents. Last but not least, annotations introduced by serialization, especially within a word, would break auto-correct and suggestions.

In order to achieve both objectives, the interactor can be designed to operate in two different modes based on the type of selected node(s):

• Insert/Delete Mode
  • Handles editing non-textual node(s) or multi-node selection.
• Text Editing Mode.
  • Handles native features when editing a single text node.

Each mode is discussed in more details below.

Insert/Delete mode only

In short:

• Handles any selection that isn't a single TextNode.
• All selection changes are handled by the DocumentInteractorState.
• Allows inserting characters over a non-textual node or multi-node selection, or deleting that selection or node(s).

In this mode, when the selection isn't a single TextNode, the soft keyboard is treated as a regular keyboard. This can be done by reseting the remote TextEditingValue as an empty-value* after each event for the given condition. As a result, whenever the user presses a key, the event is captured -- i.e. insert character(s), insert newline or backspace. Each of the three events is converted into an EditorCommand and then is sent to the DocumentEditor which in turn updates the document and selection accordingly. If the resulting selection from the command isn't a single TextNode, the remote value is reset again to an empty-value* . Otherwise, Text Editing Mode is activated.

* Technically, the value isn't empty since it'll contain the zwsp character with a selection offset = 1 which assists in detecting the backspace event.

Note: it was observed in both iOS and Android that auto-correct and suggestions are not presented when more than one word is selected. For this reason, multiple TextNode selection will be treated as non-textual nodes to avoid introducing more complexity for no added value. The user will still be able to select multiple text nodes and insert a character or multiple characters at once (e.g. swipe to text).

Text Editing Mode

In short:

• Handles any selection that is a single TextNode only.
• Selection within the selected TextNode can be changed remotely, but they're applied by the DocumentInteractorState.
• Allows editing the text with the native features.

In this mode, the remote TextEditingValue is set to contain the zwsp character at offset 0 + the selected TextNode's text. Delegating the entire node's text to be edited by the system is necessary. The system needs to know the surrounding text of a selection so that auto-correct, suggestion and voice-dictation work properly.

The main challenge here is to sync any remote changes to the selected TextNode. This is a challenge because the remote value can have multiple changes in a single event. For instance, when a user selects an auto-correct suggestion, the TextInputClient will receive a single event even if more than one character was modified by the correction (added or deleted or both). Similarly, voice dictation will modify an entire sentence in multiple places at once. For example, at the end of a voice dictation activity, the TextInputClient may receive a single event that modified the first character (capitalize it), replaced a word somewhere and inserted a dot at the end.

In order to reflect the remote changes into the selected TextNode, three steps are taken:

• Apply the remote changes to the selected TextNode (i.e. local value) by finding the differences.
• Apply appropriate attribution for any added text.
• Adjust selection if necessary.

It's worth noting that since the changes are happening within a single TextNode and from a single event, they are usually few characters. A difference with multiple characters would only occur for auto-correction, suggestion, voice dictation and swipe to text. Typically, the difference is just one character (insertion or deletion).

In the experiment mentioned in the proposal, a Dart package implementation of Google's diff-patch-match library was used to compute the difference between the remote value and the selected TextNode's text. The related code in the experiment can be seen here. The three steps mentioned earlier are also shown within the linked the method (i.e. _applyRemoteChanges).

Known Issues with this approach:

• [need to confirm] Voice dictation could stop unintentionally if the user starts the activity when the interactor is in Insert/Delete Mode (e.g. when there's a selection with multiple nodes). This will happen because once the first character is inserted, the mode will switch to Text Editing Mode which will reset the remote TextEditingValue.
• Auto-Capitalization could fail for the first character of the first sentence since the first character is the zwsp from the system point of view.

[osaxma]

To answer the questions of the proposal:

Narrow and Specialized APIs
Will all TextInput requirements make sense for Super Editor, and will Super Editor be able to accomplish all desired edits when using the available TextInput APIs?

I believe the answer is yes based on the experiment and the answer above. I haven't encountered any red flags so far and I assume the worst case scenario would be a missing platform-specific feature rather than a show stopper.

Cross-Boundary Edits
Will TextInput APIs work as expected when a selection spans multiple text content blocks?

Yes, it should work as expected as explained above and shown by the experiment. In the experiment, iOS's Notes app was used as a reference for most of the behaviors.

Multi-Media Editing
Will TextInput APIs get in the way of Super Editor's multimedia editing capabilities?

I believe multi-media editing for software keyboard is limited to deleting the multi-media node if it's selected. That can be done using backspace or inserting a character over the selected node. Any additional interaction or behavior with a multi-media node can be handed by gestures or dialogs through the interactor or the component itself. TextInput doesn't seem to pose any limitation here.

[osaxma]

This is part 2 of sharing my findings.

The following are some features that may require special considerations:

Drag Handles:

iOS	Android

• Since the selection height cannot be determined, placing the handles accurately could be a challenge.
• Color, shape and position are app or platform specific.
  • [suggestion] a Theme can be utilized here for the handles' color alongside selection highlight, caret, and cursor colors.
  • [suggestion] default shape and position can be implemented based on the platform.
• TextDirection may need to be considered for drag handle placement.
  • 

Gestures

gesture	iOS	Android
tap	places cursor on the tapped position	TBD
double tap	selects word	TBD
triple tap	selects the entire paragraph	TBD
press and hold on document	activates floating cursor	TBD
Swipe left with Three fingers	undo	n/a
Swipe right with Three fingers	redo	n/a

not all gestures were implemented in the experiment and some weren't implemented per spec.

Selection Controls

While positioning a toolbar above a selected text is possible using the default editor API (as demonstrated in the package's example), implementing the following behaviors, as observed in iOS's Notes app, may require some coordination from the DocumentInteractor:

• tapping on caret shows the selection controls.
• tapping on a selection-highlight toggles (show/hide) the selection controls.
• dragging a selection hides the selection controls during the drag activity.
  • For Android, a magnifying glass is shown and the handle that is being dragged is hidden.

Floating Cursor (iOS only)

The floating cursor can be observed at the end of the attached video of the main post.

As observed in the Notes App, the floating cursor can be activated in three different ways (only the first one was implemented in the experiment):

• press and hold the space bar of the virtual keyboard.
• press and hold anywhere on the document and then drag.
  • in this case the floating cursor is shifted upward so it appears above the finger.
• drag the caret itself.

Additionally, the floating cursor exhibits the following behavior (not implemented in the experiment):

• When the floating cursor is hovering above the text, the caret is hidden.
• When the floating cursor is not above any text, the caret is shown at its landing position (e.g. at the end of line if the floating cursor is beyond that line).
• When the caret is shown (second case), it is greyed-out.

To implement the same behaviors, the DocumentInteractor may need to have the ability to hide/show the caret as well as to temporarily change its color. On the other hand, the lack of this feature would show the caret at its landing position with its default color all the time while the floating cursor is moving (the experiment's current behavior).

All floating cursor activity comes from TextInputClient.updateFloatingCursor which provides the state (start, update, end) and a relative offset of the floating cursor.

Auto Correct Prompt (iOS only)

The auto correct prompt is reported by TextInputClient.showAutocorrectionPromptRect which provides the selection offsets (start and end) of the remote TextEditingValue.

This method can be called when the selection is collapsed right after the word (typically after the user moved the caret to that position). If the word was actually a misspelling, it'll leave a wiggly underline like this after the selection has changed and the prompted word was not selected:

Additional Considerations

Decoupling Existing Commands from Keyboard Actions

There are several commands that are tightly coupled with keyboard-actions. Some commands are in fact private to document_keyboard_actions.dart such as _PasteEditorCommand. Decoupling all the commands will make them reusable for mobile (e.g. selection controls) as well as for other features in general such as dropdown menus in desktop/web.

Handling External Keyboards For Mobile Devices

Mobile devices, especially tablets, could use an external keyboard. This poses few questions that need to be investigated:

• Does the external keyboard emit events through RawKeyboardListener?
• If so, how to detect if an external keyboard is attached?

---

I believe this covers everything that needs to be mentioned based on my experiment. I haven't had the chance to experiment with the rest of the specialized APIs such as TextInputClient.performPrivateCommand. Nevertheless, EditableTextState implementation can be used as a reference.

[osaxma]

@matthew-carroll

Exactly.

Second, there's what Flutter has implemented for EditText via TextInputClient, which may or may not match those default platform behaviors.

This is one of the main reasons I didn't rely on EditableText as a reference for the behavior (e.g. inverting a selection by dragging in the opposite direction).

Do you happen to have a regular iOS and Android app sitting around where you could screenshot behaviors like I did so we can see the platform default behaviors for a single text field?

I'm not sure what would be a regular app. Though, do you think the System's text fields (e.g. search-bar in settings or contacts) would be a good reference for the default behaviors of a single text field?

[matthew-carroll]

A "regular" app would be an Activity with an EditText on Android and a ViewController with a UITextField on iOS.

Any published app might have custom behavior, so we'll wanna figure out what happens when there's no customization.

[matthew-carroll]

I created a regular Android project with an EditText and I'm observing the following behaviors:

Collapsed selection:

• when the user taps on text in the field, the caret is placed exactly where the user taps (not the beginning or end of a word)
• when the user taps to insert a caret, the caret initially includes a drag handle under it
• when the user doesn't interact with the text for a couple seconds, the drag handle fades out
• when the user taps the drag handle and drags, the caret moves with the user's finger
• when the user drags the caret around, either by drag handle or by tapping and dragging on the text, a magnifier appears
• when the user taps and drags with initial left/right motion anywhere on the text, the caret moves with the finger, however
  • when the user taps and drags up/down instead of left/right, the user creates an expanded selection, and that selection does not display drag handles while dragging

Expanded selection:

• double tap selects an entire word. If the user double taps between words, the word closest to the tap is selected
• an expanded selection displays 2 drag handles on the selection and displays a toolbar when the user is not actively dragging
• similar to the Flutter version, when I drag left/right, I cannot cross one drag handle past the other. The selection stops moving at a single character.
• when the user drags a selection handle, the toolbar disappears and a magnifier appears
• when the user drags a selection handle, the selection expands in the direction of the handle by the word, but when the user reverses direction with that handle, text is deselected by the letter
• when the user taps and drags inside or outside of a text selection, the selection disappears and a collapsed caret moves with the finger

Demo:

Screen.Recording.2021-06-23.at.2.12.44.PM.mov

[matthew-carroll]

I created a regular iOS project with a TextField and TextEditor and I'm observing the following behaviors:

Collapsed selection:

• when the user taps on text, the caret is initially positioned at the beginning or the end of the word. The caret almost always chooses the end of the word. If the user taps very close to the beginning of the word, the beginning is chosen. I'm not sure how the decision is made.
• when the user presses and pauses for a brief moment, the caret moves to the exact text position where the user tapped and then the user can drag the caret anywhere.
  • when the user taps and immediately drags, nothing happens. It doesn't insert the caret or move the caret. It doesn't even give focus to an unfocused TextField or TextEditor
• when the user taps on the collapsed caret, a toolbar appears

Expanded selection:

• when the user double taps text, a whole word is selected
  • when double tapping between words, the nearest word is selected
• when the user triple taps...
  • TextField: all text is selected
  • TextEditor: a paragraph is selected
• when the selection is expanded, a toolbar is visible
  • when the user drags a drag handle, the toolbar disappears
  • when the user taps on an expanded selection, the toolbar display toggles on/off
• when the user drags one drag handle past another, there is a drag-handle handoff where the user ends up dragging the other handle. For example, if the user drags the trailing handle to the left and then passes the leading handle, suddenly the user is dragging the leading handle, and vis-a-versa.
  • when the user drags one handle directly over another handle, both handles occupy the same position
    • when the user stops dragging a drag handle and leaves it on top of the other drag handle, the selection collapses and behaviors like any other collapsed selection

Keyboard floating cursor:

• the TextEditor behaves as expected. The floating cursor is confined to the editor area and operates like the Flutter version.
  • the floating cursor is very tall
• the TextField...
  • the floating cursor is the height of the regular cursor
  • allows the floating cursor to move above or below the visible area of the textfield, but continues to map the text position with the gray cursor
  • moving the caret beyond the visible text results in an auto-scroll, but this behavior is a bit unnatural because there is limited horizontal space on the keyboard, while the text in the TextField might go on for a very long time.
    • Once the TextField begins autoscrolling, it keeps autoscrolling in that direction as long as the user doesn't move their finger on the keyboard. The TextField will even scroll so far that the caret is no longer visible.

Screen.Recording.2021-06-23.at.2.48.47.PM.mov

[matthew-carroll]

@osaxma please let me know if you have any thoughts or observations WRT these default platform examples.

[matthew-carroll]

@osaxma do you happen to know how to accomplish the magnifying glass? I started looking into that back in 2018 but then iOS announced they were removing it so I stopped. I think it might use some kind of Layer interaction with the compositor, but I'm not sure.

[matthew-carroll]

I have the magnifying glass working now.

[knopp]

There is a good reason to use TextInput on desktop as well. Certain things, such as text composition or user interface for selecting accents will not work without. Also right now dead keys do not seem to be working at all.

It would also be good to use DefaultTextEditingActions, as ideally on platforms where supported (macOS) those should be initiated driven platform itself (NSStandardKeyBindingResponding on macOS). But IIRC this is still very much work in progress in Flutter.

Edit: I think this is the flutter umbrella issue for it.