Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issues related to keyboard inset #37719

Merged
merged 17 commits into from
Dec 14, 2022
Merged

Conversation

vashworth
Copy link
Contributor

@vashworth vashworth commented Nov 17, 2022

Refactors keyboard logic to use a combination of UIKeyboardWillShowNotification, UIKeyboardWillChangeFrameNotification, UIKeyboardWillHideNotification notifications to determine the keyboard inset.

When a docked keyboard is shown, or when it changes from undocked/floating to docked, UIKeyboardWillShowNotification is triggered. And when a keyboard is hidden or changes from docked to undocked/floating, UIKeyboardWillHideNotification is triggered. So it would make sense to calculate and set the inset when UIKeyboardWillShowNotification happens and set the inset to 0 when UIKeyboardWillHideNotification happens.
However, there are some cases where these events are missing, have incomplete values, or do something unexpected, so we use UIKeyboardWillChangeFrameNotification in addition to help handle those cases.

Note: This fix makes a best effort to accommodate Stage Manger, but does not fix all issues related to Stage Manager. In addition, it does not address issues related to the inset when the keyboard is open and the screen orientation is rotated.

Fixes flutter/flutter#86847.
Fixes flutter/flutter#104689.
Fixes flutter/flutter#109845.
Fixes flutter/flutter#115528.
Fixes flutter/flutter#53565.

Issue Before After
flutter/flutter#86847 keyboard-floating-bug keyboard-floating-fix
flutter/flutter#104689 keyboard-undocked-bug keyboard-undocked-fix
flutter/flutter#109845 keyboard-splitview-bug keyboard-splitview-fix
flutter/flutter#115528, flutter/flutter#53565 keyboard-iphone-bug keyboard-iphone-fix
flutter/flutter#115528, flutter/flutter#53565 keyboard-iphone-bug-2 keyboard-iphone-fix-2

Other use cases I tested manually. I tested each case in Full Screen View, Split View, Slide Over View. I tested each case with a custom keyboard with a floating decimal height (see Custom keyboard extension code sample below).

  • Docked Keyboard
    • Opening/Closing
    • Dragging docked keyboard and then putting back
    • Rotating orientation while keyboard open
  • Undocked Keyboard (iPad only)
    • Changing from docked to undocked via menu
    • Opening/Closing
    • Dragging to a new position
    • Changing from undocked to docked via menu
    • Dragging from undocked to docked
    • Changing from undocked to split via menu
    • Changing from undocked to floating via menu
    • Pinched from undocked to floating
    • Changing from undocked to expanded shortcuts bar via menu button
    • Rotating orientation while keyboard open
  • Split Keyboard (iPad only)
    • Changing from docked to split via menu
    • Dragged from docked to split
    • Opening/Closing
    • Dragging to a new position
    • Changing from split to docked via menu
    • Dragging from split to docked
    • Changing from split to undocked via menu
    • Pinching from split to undocked
    • Dragging from split to split docked
    • Changing from split to floating via menu
    • Changing from split to expanded shortcuts bar via menu button
    • Rotating orientation while keyboard open
  • Floating Keyboard (iPad only)
    • Changing from docked to floating via menu
    • Pinching from docked to floating
    • Opening/Closing
    • Dragging floating to docked
    • Spreading floating to docked
    • Rotating orientation while keyboard open
  • Expanded Shortcuts Bar (iPad only)
    • Changing from docked to expanded shortcuts bar via menu button
    • Changing from minimized shortcuts bar to expanded shortcuts bar via menu button
    • Opening/Closing
    • Dragging from expanded shortcuts bar to minimized shortcuts bar
    • Rotating orientation while keyboard open
  • Minimized Shortcuts Bar (iPad only)
    • Changing from expanded shortcuts bar to minimized shortcuts bar via menu
    • Opening/Closing
    • Dragging to a new position
    • Dragging from minimized shortcuts bar to expanded shortcuts bar
    • Rotating orientation while keyboard open
  • One-handed (iPhone only)
    • Changing from docked to one-handed right
    • Changing from docked to one-handed left
    • Opening/Closing
    • Changing from one-handed right to docked
    • Changing from one-handed left to docked
  • Switching between states
    • Open original app, switch to second app, open keyboard, switch back to original app, open keyboard, switch to second app, switch to original app
    • Open original app, switch to second app, open keyboard, switch back to original app, switch to second app, switch to original app
    • Swipe down to open Control Center, close control center and click on input field quickly

Devices I tested with:

  • Simulator iPhone X (iOS 12.4)
  • Simulator iPhone 11 Pro (iOS 13.0)
  • Simulator iPhone 12 Pro (iOS 14.3)
  • Simulator iPhone 14 Pro (iOS 16.1)
  • Simulator iPad Pro 9.7in (iOS 12.4)
  • Simulator iPad Pro 9.7in (iOS 14.3)
  • Simulator iPad 9th gen (iOS 16.1)
  • Physical iPad Pro 11 in (iOS 16.1)

Sample code I used for the flutter app: flutter/flutter#104689 (comment)

Custom keyboard extension code sample
import UIKit

class KeyboardViewController: UIInputViewController {

    @IBOutlet var nextKeyboardButton: UIButton!
    @IBOutlet var closeKeyboardButton: UIButton!
    
    override func updateViewConstraints() {
        super.updateViewConstraints()

        let expandedHeight: CGFloat = 397 + (2/3)
        let constraint = NSLayoutConstraint(item: self.view, attribute: .height, relatedBy: .equal, toItem: nil, attribute: .notAnAttribute, multiplier: 0.0, constant: expandedHeight)
        self.view.addConstraint(constraint)
    }
    
    override func viewDidLoad() {
        super.viewDidLoad()
        
        self.nextKeyboardButton = UIButton(type: .system)
        self.nextKeyboardButton.setTitle(NSLocalizedString("Next Keyboard", comment: "Title for 'Next Keyboard' button"), for: [])
        self.nextKeyboardButton.sizeToFit()
        self.nextKeyboardButton.translatesAutoresizingMaskIntoConstraints = false
        self.nextKeyboardButton.addTarget(self, action: #selector(handleInputModeList(from:with:)), for: .allTouchEvents)
        self.view.addSubview(self.nextKeyboardButton)
        self.nextKeyboardButton.leftAnchor.constraint(equalTo: self.view.leftAnchor).isActive = true
        self.nextKeyboardButton.bottomAnchor.constraint(equalTo: self.view.bottomAnchor).isActive = true
        
        self.closeKeyboardButton = UIButton(type: .system)
        self.closeKeyboardButton.setTitle(NSLocalizedString("Close Keyboard", comment: "Title for 'Close Keyboard' button"), for: [])
        self.closeKeyboardButton.sizeToFit()
        self.closeKeyboardButton.translatesAutoresizingMaskIntoConstraints = false
        self.closeKeyboardButton.addTarget(self, action: #selector(closeKeyboard), for: .allTouchEvents)
        self.view.addSubview(self.closeKeyboardButton)
        self.closeKeyboardButton.rightAnchor.constraint(equalTo: self.view.rightAnchor).isActive = true
        self.closeKeyboardButton.bottomAnchor.constraint(equalTo: self.view.bottomAnchor).isActive = true
    }
    
    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)
    }
    
    override func viewWillLayoutSubviews() {
        self.nextKeyboardButton.isHidden = !self.needsInputModeSwitchKey
        super.viewWillLayoutSubviews()
    }
    
    @objc func closeKeyboard() {
        self.textDocumentProxy.insertText("\n")
    }
}

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
  • I listed at least one issue that this PR fixes in the description above.
  • I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
  • I updated/added relevant documentation (doc comments with ///).
  • I signed the CLA.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@vashworth vashworth marked this pull request as ready for review November 18, 2022 20:17
@jmagman
Copy link
Member

jmagman commented Nov 18, 2022

Adding @cyanglaz.
@luckysmg may have input.

@luckysmg
Copy link
Contributor

luckysmg commented Nov 19, 2022

Thx @vashworth for working on fixing this.

(Just some advices and sharing my thoughts, maybe not very appropriate😄).

I think now we will deal with different situtation in keyboard notification function, and we use some judgement to judge the keyboard is docked or undocked or floating. This is ok, emm but maybe not very clear, and it is not very easy to know current judgement branch will impact which scenario...

I think maybe we can do sth to make this more clear like this

  1. In this scenario,we can define an class or enum, maybe named with: KeyboardAttachMode
  2. And we compute a KeyboardAttachMode using the NSNotification passed by keyboard notification(where we can get the keyboard frame from)
  3. And final we can calculate the end frame by using the keyboard attach mode and notificaition.

Here is the some demo code.

Enum KeyboardAttachMode {
   docked,
   undocked,
   floating
   .......
}

- (KeyboardAttachMode)computeKeyboardAttachMode:(NSNotification noti){
       CGRect  endFrame = noti["UIKeyboardEndFrameUserKey"]
       if (endFrame xxxxxx) {
           return KeyboardAttachMode.docked;
      } else if (xxxx) {
           return KeyboardAttachMode.floating;
      }
}

- (double)computeInset(KeyboardAttachMode mode, NSNotification notification) {
   double inset = 0;
   if (mode == .docked) {
        inset = xxxxxx
   } else mode == .floating{
   } else .....{
  }
  return inset;
}

- (void)keyboardShow {
    KeyboardAttachMode mode = computeKeyboardAttachMode(noti);
    double computeInset = computeInset(mode,noti);
     if (computeInset != self.targetButtomInset) {
        self.targetBottomInset = computeInset;
        startKeyboardAnimation();
     }
}

- (void)keyboardWillChangeFrame {
    KeyboardAttachMode mode = computeKeyboardAttachMode(noti);
    double computeInset = computeInset(mode,noti);
     if (computeInset != self.targetButtomInset) {
        self.targetBottomInset = computeInset;
        startKeyboardAnimation();
     }
}

- (void)keyboardWillHide {
    KeyboardAttachMode mode = computeKeyboardAttachMode(noti);
    double computeInset = computeInset(mode,noti);
     if (computeInset != self.targetButtomInset) {
        self.targetBottomInset = computeInset;
        startKeyboardAnimation();
     }
}

I think this will be more clear O(∩_∩)O

Copy link
Contributor

@cyanglaz cyanglaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to use scenario app test to do some UITests? Maybe have a UITextField as PlatformView to open up keyboard, undock the keyboard and take a screenshot?

I think you mentioned this in the team meeting, is it not possible because there is no way to make the keyboard undock in XCUITests?

- (void)keyboardWillChangeFrame:(NSNotification*)notification {
// Immediately prior to a change in keyboard frame, this notification is triggered.
// There are some cases where UIKeyboardWillShowNotification & UIKeyboardWillHideNotification
// do not act as expected and this is used to catch those cases.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the change frame notification work in all cases? Maybe we should eliminate UIKeyboardWillShowNotification and UIKeyboardWillHideNotification if the change frame notification is more reliable?

Could be a nice refactor in a follow up PR if we can use frame change notification for all cases.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So none of them are very reliable which is why I used all three of them. Here's some of the use cases for example

  • When predictive-only/minimized keyboard is dragged, it sends a UIKeyboardWillShowNotification notification where the keyboard = CGRectZero, which we use to determine it's "floating" and should have inset 0 . Whereas in UIKeyboardWillChangeFrameNotification we skip when keyboard = CGRectZero, because otherwise it will set inset to 0 prematurely in some cases like when you dragged a docked keyboard but don't actually change it from docked.
  • When keyboard is changed from docked to floating, it's supposed to send a UIKeyboardWillHideNotification notification but it doesn't, so we handle in UIKeyboardWillChangeFrameNotification instead
  • When keyboard goes from docked to undocked, the UIKeyboardWillChangeFrameNotification position will not be completely below the screen so it will still have an inset, which is at least one reason why we also need UIKeyboardWillHideNotification

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, gotcha. I think logic this complex deserves a design documentation :) And we should have a comment in the code to link to the design doc. (The same way PlatformView should have and never did :p )

I think for this PR, it would be nice to explain "some cases" a little more in the comments. Or add a link in the code to your comment here: #37719 (comment)

@cyanglaz
Copy link
Contributor

However, there are some cases where these events are missing, have incomplete values, or do something unexpected, so we use UIKeyboardWillShowNotification in addition to help handle those cases.

Did you mean "UIKeyboardWillChangeFrameNotification"? :)

@jmagman
Copy link
Member

jmagman commented Dec 5, 2022

Failing tests:
	-[FlutterTextInputPluginTest testInputViewsHasNonNilInputDelegate]
	-[FlutterViewControllerTest testCalculateKeyboardInset]

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8795849902472179793/+/u/test:_Host_Tests_for_ios_debug_sim__3_/stdout

@jmagman
Copy link
Member

jmagman commented Dec 7, 2022

Nice, tests now passing! Thanks for tracking those notification issues @vashworth.

Copy link
Member

@jmagman jmagman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@luckysmg thanks so much for the thorough review on this one! Now that you have contributor permissions 🎉 can you mark when it's up to your approval standards?

@vashworth let's hold off merging this until that review is complete.

@luckysmg
Copy link
Contributor

luckysmg commented Dec 8, 2022

Oh yeah. I will have a final check and manual locally test this today (^▽^).

Copy link
Contributor

@hellohuanlin hellohuanlin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few questions.

Your design doc is really helpful for me to understand the code. It makes the PR review a lot easier! Great work!

std::unique_ptr<fml::WeakPtrFactory<FlutterEngine>> _weakFactory =
std::make_unique<fml::WeakPtrFactory<FlutterEngine>>(engine);
XCTestExpectation* invokeExpectation =
[self expectationWithDescription:@"isLiveTextInputAvailableInvoke"];
FlutterPlatformPlugin* plugin =
[[FlutterPlatformPlugin alloc] initWithEngine:_weakFactory->GetWeakPtr()];
[[[FlutterPlatformPlugin alloc] initWithEngine:_weakFactory->GetWeakPtr()] autorelease];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for fixing all of these.

}

// Ignore keyboard notifications if engine’s viewController is not current viewController.
// Engine’s viewController is not current viewController.
if ([_engine.get() viewController] != self) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this happen when an app shows multiple flutter view controller on the same screen?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, from my understanding this can happen when you're using add-to-app stuff like this flutter/flutter#39036 (comment). This is a good point, though. I didn't test use case like that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: I did test this use case manually and everything appeared to be working correctly.

@luckysmg
Copy link
Contributor

Mac mac_ios_engine_release failed, maybe need to rebase to main branch ?

Copy link
Contributor

@luckysmg luckysmg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM O(∩_∩)O

Copy link
Contributor

@cyanglaz cyanglaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Member

@jmagman jmagman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for tracking all these corner cases down! LGTM!

Copy link
Contributor

@hellohuanlin hellohuanlin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making those changes. Your design doc is very helpful reviewing the PR.

@vashworth vashworth added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 14, 2022
@auto-submit auto-submit bot merged commit 479bb73 into flutter:main Dec 14, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Dec 14, 2022
auto-submit bot pushed a commit to flutter/flutter that referenced this pull request Dec 15, 2022
…117109)

* 1a1b1feee Roll Skia from 537e1e8c1ca6 to 729ccbfb87bc (7 revisions) (flutter/engine#38277)

* 3b18821e1 Roll Fuchsia Linux SDK from A0jnUUORf2LQu1z2V... to e2lfUFBW5ddtTZBbw... (flutter/engine#38280)

* beb94ea2c Roll Skia from 729ccbfb87bc to 3171deabd88a (4 revisions) (flutter/engine#38279)

* 8f9071ab9 Roll Fuchsia Mac SDK from FQQdl8AGAsALFniHl... to u-tC0QEGUT4xQ4KOo... (flutter/engine#38282)

* aa78cd8d6 add link to website (flutter/engine#38273)

* 955447b35 pylint all Python scripts under testing/ (flutter/engine#38268)

* 3f22e1958 [web] correct float count in runtime effect (flutter/engine#38288)

* 479bb736f Fix issues related to keyboard inset (flutter/engine#37719)

* 6cd85616b [macOS] Refactor rendering infrastructure (flutter/engine#37789)

* d1533d12b [web] Make Canvaskit's malloc more useful (flutter/engine#38130)

* db5605ea7 Fix new `unnecessary_parenthesis` diagnostics. (flutter/engine#38291)
loic-sharma pushed a commit to loic-sharma/flutter-engine that referenced this pull request Dec 16, 2022
* fix keyboard inset not collapsing when expected

* fix some formatting

* fix issue with rotating with undocked and split keyboard

* fix formatting

* fix behavior on slide over view

* fix formatting

* refactor to make logic more clear

* move enum to header file, remove unneeded parameters, syntax fixes, remove rotation logic

* ignore notification if app state is not active, change way it checks if keyboard intersects with screen to accomodate for repeating decimals, format

* fix leaking unit test

* use viewIfLoaded and update tests to fix mocking

* change ignore logic related to application state to be more specific

* add more comments

* add more comments

* change function name to be more clear, add warning log if view is not loaded, update a comment
loic-sharma pushed a commit to loic-sharma/flutter-engine that referenced this pull request Jan 3, 2023
* fix keyboard inset not collapsing when expected

* fix some formatting

* fix issue with rotating with undocked and split keyboard

* fix formatting

* fix behavior on slide over view

* fix formatting

* refactor to make logic more clear

* move enum to header file, remove unneeded parameters, syntax fixes, remove rotation logic

* ignore notification if app state is not active, change way it checks if keyboard intersects with screen to accomodate for repeating decimals, format

* fix leaking unit test

* use viewIfLoaded and update tests to fix mocking

* change ignore logic related to application state to be more specific

* add more comments

* add more comments

* change function name to be more clear, add warning log if view is not loaded, update a comment
gspencergoog pushed a commit to gspencergoog/flutter that referenced this pull request Jan 19, 2023
…lutter#117109)

* 1a1b1feee Roll Skia from 537e1e8c1ca6 to 729ccbfb87bc (7 revisions) (flutter/engine#38277)

* 3b18821e1 Roll Fuchsia Linux SDK from A0jnUUORf2LQu1z2V... to e2lfUFBW5ddtTZBbw... (flutter/engine#38280)

* beb94ea2c Roll Skia from 729ccbfb87bc to 3171deabd88a (4 revisions) (flutter/engine#38279)

* 8f9071ab9 Roll Fuchsia Mac SDK from FQQdl8AGAsALFniHl... to u-tC0QEGUT4xQ4KOo... (flutter/engine#38282)

* aa78cd8d6 add link to website (flutter/engine#38273)

* 955447b35 pylint all Python scripts under testing/ (flutter/engine#38268)

* 3f22e1958 [web] correct float count in runtime effect (flutter/engine#38288)

* 479bb736f Fix issues related to keyboard inset (flutter/engine#37719)

* 6cd85616b [macOS] Refactor rendering infrastructure (flutter/engine#37789)

* d1533d12b [web] Make Canvaskit's malloc more useful (flutter/engine#38130)

* db5605ea7 Fix new `unnecessary_parenthesis` diagnostics. (flutter/engine#38291)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
autosubmit Merge PR when tree becomes green via auto submit App platform-ios
Projects
None yet
5 participants