TextField should support apple pencil input by default.

[softmarshmallow]

related: #38135

not the complex implementation for pressure, tilt, etc.
but textfield should support apple pencil input by default.

figure 1 - text recignition on default text inputs.

more info about this feature

apple pencil text input works on web html inputs by default, i think it just needs to be recognized as input from ios.

Great thanks to the flutter team.

[fbcouch]

Does anyone have any thoughts and/or direction on this?

I took a crack at implementing UIIndirectScribbleInteractionDelegate to see how that might work. The upshot is, I can get text that users write, but it doesn't support all of the scribble interactions like scratching out text to erase it since I'm not giving the iPadOS an actual representation of the text in order for it to know what word(s) to erase.

I'm imagining that we will need some kind of custom UITextField-ish representation of the EditableText from flutter.

[fbcouch]

@cbracken Your name is all over some TODOs in the fields I've been messing with trying to figure out how to implement this, so figured I would tag you to see if you have any thoughts.

I have been playing around in FlutterTextInputPlugin.mm:
twinsunllc/engine@c5664a8

So far, it seems that listening for the TextInput.setEditableSizeAndTransform method call and setting the frame of the _activeView and implementing - (CGRect)firstRectForRange:(UITextRange*)range so that it returns a rect roughly in the area of the given range are the minimum needed.

There are still some funny issues (and you can't start scribbling till the field is focused), but that seems like a starting point. Do you have any thoughts on the right way to go about implementing firstRectForRange? I imagine that needs to accurately transform the given range into a rectangle matching where that text is on the screen.

Tagging @justinmc and @LongCatIsLooong as well since you have recent commits in that file.

[justinmc]

CC @xster in case you have any existing plans for Apple Pencil support.

Do we have any existing text code that would help like this?https://github.com/flutter/engine/blob/63a78df571e88e5bf715645afb2a6b0fbf510706/lib/ui/text/paragraph.cc#L118

It would be pretty cool if we could get this working. Thanks @fbcouch for investigating.

[fbcouch]

@justinmc Thanks for pointing out that code. That actually helped me make a little more progress on this.

I couldn't figure out how to get access to the Paragraph objects from FlutterTextInput, so what I've done is add some glue code on the flutter side (meaning, in the flutter/flutter repo, not sure how you refer to this): twinsunllc@7b39621

That sends the selection boxes through the channel, and then used those on the engine side to implement some of the relevant methods: https://github.com/twinsunllc/engine/tree/feature/scribble

This kinda works, but there are still a few issues that I may need some help figuring out:

• The FlutterTextInput having a size seems to "consume" tap events so that the cursor can't be moved around with non-scribble methods (instead of being able to tap with the pencil or a finger, you have to draw a short line with the pencil similar to how you can make a line through a word to select it). I tried forwarding the touchesBegan, etc selectors to the superview, but that didn't help.
• There do seem to be some issues still remaining around managing the selected text. Somehow, I keep ending up with the last character selected in addition to whatever actual selection is there.
• Once text is selected, making any marks will (usually) replace that text rather than making a new selection. I suspect that is related to the previous item. Possibly some kind of issue keeping the FlutterTextInput selection synced up with the flutter side?
• Finally, scribble only works once the input is focused, which is not how it works natively

I'm going to ask about this on the discord channel, but figured it would be good to have my findings recorded here as well.

[justinmc]

Sounds like you're getting somewhere. +1 to asking on Discord. You also might consider opening a draft PR to make it easier to take a look at your code.

[LongCatIsLooong]

The FlutterTextInput having a size seems to "consume" tap events

iirc currently FlutterTextInputView isn't expecting any touch events. Touch events are forwarded in the FlutterViewController: https://github.com/flutter/engine/blob/1411cb671be8ddf4f16f8d3bd96d7a86a43b6fd3/shell/platform/darwin/ios/framework/Source/FlutterViewController.mm#L934-L948

Once text is selected, making any marks will (usually) replace that text rather than making a new selection.

I think
- [FlutterTextInputView setMarkedText:(NSString*)markedText selectedRange:(NSRange)]
does this so multistage input replaces the current selection:
https://github.com/flutter/engine/blob/1411cb671be8ddf4f16f8d3bd96d7a86a43b6fd3/shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm#L759-L767

scribble only works once the input is focused

That might be because we try to keep as few FlutterTextInputs around as possible. iirc if a text field gets unfocus and it has nothing to do with username or password (for autofill) then the corresponding FlutterTextInput gets removed from the view hierarchy, to prevent leaks.

[fbcouch]

Thanks @LongCatIsLooong that is helpful – I pushed a couple of commits to that PR using UIScribbleInteractionDelegate to disable setMarkedText during scribbling, and pass a reference to the FlutterViewController to the FlutterTextInput to forward touch events. That seems to have fixed the issues with touch events and scribble-to-select replacing the current selection. Disabling the showAutocorrectionPromptRectForStart within firstRectForRange while scribble is in progress also got rid of the issue where the last character was already selected.

So, I think the remaining issues here are:

• This only works after focusing on the input
• The selection doesn't bring up the selection handles

[fbcouch]

Between twinsunllc/engine#1 and twinsunllc#1 I have the selection handles working properly now, so the last thing is starting the interaction without having focus already. I think the right thing to do will be using UIIndirectScribbleInteractionDelegate and trying to pass focus to the correct element...fortunately func indirectScribbleInteraction(UIInteraction, focusElementIfNeeded: Self.ElementIdentifier, referencePoint: CGPoint, completion: ((UIResponder & UITextInput)?) -> Void) has that completion handler, so I assume it is not on the UI thread and I could wait for a response to a message over a flutter channel. We will see, anyway.

[fbcouch]

Yep, that approach seems to have worked. I have updated the draft PRs – I'm going to ask in discord to see if I can get some kind of preliminary review of the approach before rebasing onto master/adding tests (which I assume is necessary to open a real PR)

[justinmc]

The engine PR has been merged and the framework PR is waiting for final review, then we can ship this.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.