[SuperTextField] Flutter keyboard bugs workarounds and easier mapping shortcuts to keyboard actions

[venkatd]

Overview

Implementing TextFieldKeyboardHandler is not user friendly for two reasons:

1. RawKeyEvent and RawKeyboard.instance.keysPressed have many open bugs in Flutter (listed below)
2. Mapping keyboard shortcut combinations to handlers is tedious.

The proposed changes would be:

• TextFieldKeyboardHandler would take LogicalKeyEvent (or something like it) as a parameter rather than RawKeyEvent. The event would protect SuperTextField devs from the messy details of Flutter bugs. (All of these bugs have been open for months, many with no activity, so don't think they will get closed any time soon.)

• Expose a ShortcutsHandler to make mapping keyboard combinations to TextFieldKeyboardHandler easier, similar to how in Flutter we can map shortcuts to intents.

Here is a gist of the relevant code we have in our own app:
https://gist.github.com/venkatd/dcdefb929206b73ccb3d73b955d80f0c#file-logical_key_event-dart

I currently convert a RawKeyEvent to a LogicalKeyEvent in each handler. In this scenario below logicalKeyEvent.character just works out of the box since the workarounds are internal to LogicalKeyEvent. I can access the character and decide if I want to insert it or not.

TextFieldKeyboardHandlerResult insertTypedCharacter({
  required AttributedTextEditingController controller,
  required SuperSelectableTextState selectableTextState,
  required RawKeyEvent keyEvent,
}) {
  final logicalKeyEvent = LogicalKeyEvent.fromRawKeyEvent(
    keyEvent,
    RawKeyboard.instance.keysPressed,
  );

  final char = logicalKeyEvent.character;
  if (char == null) return TextFieldKeyboardHandlerResult.notHandled;

  controller.execute(ReplaceSelectedText(char));

  return TextFieldKeyboardHandlerResult.handled;
}

A dev can add their own key bindings earlier in the list of actions to override default shortcuts:

final myHandlers = <TextFieldKeyboardHandler>[
  ShortcutsHandler({
    LogicalKeySet(LogicalKeyboardKey.enter, LogicalKeyboardKey.shift,
        LogicalKeyboardKey.control): ReplaceSelectedText('\n---\n'),
    LogicalKeySet(LogicalKeyboardKey.enter, LogicalKeyboardKey.shift):
        ReplaceSelectedText('\n'),
  }),
  ...kPlatformKeyboardHandlers,
];

LogicalKeyEvent is also used in the shortcuts implementation so I can look up the matching key combination from a map (https://gist.github.com/venkatd/dcdefb929206b73ccb3d73b955d80f0c#file-keyboard_handlers-dart-L170-L186)

Perhaps renaming LogicalKeyEvent to TextFieldKeyboardEvent` could make the intent clearer?

Related Flutter PRs/issues

ShortcutActivator PR: flutter/flutter#78522

This is specialized to the Shortcuts/Intents/Actions, while LogicalKeyEvent takes a similar approach to remove OS keyboard combinations.

The "WHY" section in the PR explains the gotcha's on how we would want to map keyboard shortcuts to actions.

If I want to maintain a map of a KeySet to a handler and properly trigger the handlers, I need to account for the edge cases explained in "WHY".

"Sticky" modifier keys on Flutter web:
flutter/flutter#75627 (comment)
(Apparently other platforms too, but I could only repro on web)

Flutter web reports UI keys
flutter/flutter#81940
We're handling this on the keyboard handler side, but I've encapsulated this filter so any dev writing their own action doens't have to think about this edge case.

MacOS, if a keyboard shortcut combo includes the shift key, the letter becomes upper-cased so doesn't match:
flutter/flutter#75595

Finally, there's an issue related to a quote key on international keyboard layouts, but I think this would be SuperTextField specific since using a TextInputChannel would resolve the issue. Filing it inside this repo for now:
#244

[matthew-carroll]

The goal of this ticket is to ensure that keyboard handlers in the package don't trigger when keys beyond the desired keys are pressed.

Example:
cmd+a => select all
cmd+shift+a => do not select all

[venkatd]

@matthew-carroll

Here is the code in Flutter that checks if a key is pressed

https://github.com/flutter/flutter/blob/6e08ea0a2cc0985c7ebc0acae83f3e780dd67499/packages/flutter/lib/src/widgets/shortcuts.dart#L305-L311

Of note is that, the RawKeyEvent is almost not used at all. It just checks that it's of type RawKeyDownEvent. But the real data comes from RawKeyboard.instance.keysPressed. RawKeyboard.instance is the 2nd argument to accepts.

Actually when you check keyEvent.isMetaPressed, it is really just calling the underlying global instance RawKeyboard.instance.keysPressed.

https://github.com/flutter/flutter/blob/6e08ea0a2cc0985c7ebc0acae83f3e780dd67499/packages/flutter/lib/src/services/raw_keyboard.dart#L441-L443

... and that calls isKeyPressed which just calls out to this:

https://github.com/flutter/flutter/blob/6e08ea0a2cc0985c7ebc0acae83f3e780dd67499/packages/flutter/lib/src/services/raw_keyboard.dart#L441-L443

So it looks like all this abstraction is unnecessary. All we care about is RawKeyboard.instance.keysPressed. We might want to care about RawKeyEvent.logicalKey if we want to know the latest key, but that might not be critical for now.

It would be an edge case. For example meta+left would mean left is the final trigger while meta is the modifier. So left+meta wouldn't trigger anything.

I've pulled out a helper that is decoupled from a lot of the Flutter concepts:

keyboard.dart

import 'package:flutter/services.dart';

/// Returns true if two sets of key combinations match exactly, accounting for
/// synonyms. For example [LogicalKeyboardKey.shiftLeft] is equivalent to
/// [LogicalKeyboardKey.shift], so both would be treated the same.
bool keyComboMatches(
    Set<LogicalKeyboardKey> pressed, Set<LogicalKeyboardKey> target) {
  final pressedCollapsed = _collapseKeyboardKeySynonyms(pressed);
  final targetCollapsed = _collapseKeyboardKeySynonyms(target);
  return _setsEqual(pressedCollapsed, targetCollapsed);
}

Set<LogicalKeyboardKey> _collapseKeyboardKeySynonyms(
    Set<LogicalKeyboardKey> keys) {
  final collapsed = <LogicalKeyboardKey>{};
  for (final k in keys) {
    collapsed.add(_keyboardKeySynonyms[k] ?? k);
  }
  return collapsed;
}

final _keyboardKeySynonyms = <LogicalKeyboardKey, LogicalKeyboardKey>{
  LogicalKeyboardKey.shiftLeft: LogicalKeyboardKey.shift,
  LogicalKeyboardKey.shiftRight: LogicalKeyboardKey.shift,
  LogicalKeyboardKey.metaLeft: LogicalKeyboardKey.meta,
  LogicalKeyboardKey.metaRight: LogicalKeyboardKey.meta,
  LogicalKeyboardKey.altLeft: LogicalKeyboardKey.alt,
  LogicalKeyboardKey.altRight: LogicalKeyboardKey.alt,
  LogicalKeyboardKey.controlLeft: LogicalKeyboardKey.control,
  LogicalKeyboardKey.controlRight: LogicalKeyboardKey.control,
};

bool _setsEqual<T>(Set<T> a, Set<T> b) {
  return a.length == b.length && a.difference(b).isEmpty;
}

It can be used like so:

TextFieldKeyboardHandlerResult _insertNewLineOnShiftEnter({
  required AttributedTextEditingController controller,
  required SelectableTextState selectableTextState,
  required RawKeyEvent keyEvent,
}) {
  final keys = RawKeyboard.instance.keysPressed;

  // @TODO: May want to check that keyEvent.logicalKey == LogicalKeyboardKey.enter so things trigger in the right order
  if (!keyComboMatches(
      keys, {LogicalKeyboardKey.enter, LogicalKeyboardKey.shift})) {
    return TextFieldKeyboardHandlerResult.notHandled;
  }

  controller.insertNewline();

  return TextFieldKeyboardHandlerResult.handled;
}

I'm not sure if this is the implementation that makes sense for this project, but it's worth noting that the only inputs that seem to matter are:

• Set<LogicalKeyboardKey> RawKeyboard.instance.keysPressed: the keys currently pressed down
• keyEvent.logicalKey the most recent key that was just pressed. (Not 100% necessary, but later can be used to make it more robust by making the last character the trigger.)
• keyEvent.character you're using this to enter characters into

Doing things like checking isMetaPressed or isShiftPressed is really just looking at RawKeyboard.instance.keysPressed underneath.

[venkatd]

Btw this may require further investigation. There's a recent PR on the Flutter team which seems to address edge cases related to keyboard shortcuts:

flutter/flutter#78522

It's worth reading through their design decisions and motivations.

[venkatd]

I've upgraded our Flutter implementation to the master branch and changed how we're managing shortcuts. We've got some workarounds based on some Flutter bugs around RawKeyboard.instance.keyPressed. I can share how we're managing it in our app on the next catch up.

[matthew-carroll]

@venkatd can you update this ticket with the latest description of the specific issue at hand?

[venkatd]

@matthew-carroll I'll update this issue early next week.

There are a handful of issues we ran into, most of which were Flutter bugs and some specific to Flutter web. Since we've resolved all of our keyboard issues internally, it might be easiest to share our solution with comments explaining each workaround.

I introduced a LogicalKeyEvent which encapsulates the ugliness of the workarounds and makes it easier to bind key combinations to actions.

Longer term I think these issues need to be addressed by the Flutter team.

[matthew-carroll]

Thanks for the updated information.

For the problems you listed under "Bugs in how Flutter handles keyboard shortcuts", can you provide concrete examples of each issue. It's not clear what the problem is for some of them.

Also, do each of those issues have tickets in the Flutter project? If not, would you mind filing them? And can you add links to those issues in your comment so that people who find this thread can easily make their way to the Flutter tickets?

With regard to super_editor, I'm wondering what the desired action items are for this ticket. I see that you've described some approaches you took in your app. Are there specific changes that you think should be made to SuperEditor and/or SuperTextField?

[venkatd]

I've updated the main description of this issue with the latest and deleted my comment

[matthew-carroll]

I'm closing this after discussion with @venkatd - he'll open another issue with a focus on improved shortcut support.