Pass Android Q insets.systemGestureInsets to Window #10413

shihaohong · 2019-08-01T18:38:11Z

Description:

This passes Android Q's insets.systemGestureInsets to the Window. This becomes necessary as Flutter app developers want to take advantage of their entire screen with full gestural navigation mode coming with Q.

Edit: Since unit testing for Android Q involves a lot of unknowns and issues (see flutter/flutter#38471), we're going to add tests as a follow up to this PR once a solution is found

Issues:

Related to flutter/flutter#34678, flutter/flutter#34678

Todos:

Java unit tests (edit: see Engine JUnit tests don't support newer SDKs flutter#38471)

Not able to properly retrieve systemGestureInsets yet

chinmaygarde

Have a few queries that would be great to have clarified in docstrings:

How do the view and gesture insets interact? Does it matter if the system puts gestures recognizers vs window decorations in that area? Not sure it does. An argument could be made that the Flutter application author should not put interactive controls in the gesture insets but is safe to do so within view insets. But the view insets have been shown to be accommodations for status bars and such (which are gesture sensitive). If there is no difference, then the gesture insets could be specified in the current view insets.

shihaohong · 2019-08-01T22:51:04Z

The main reason they're kept separate is that I feel that the view and gesture insets feel like orthogonal use cases.

For example, there will be system gesture insets on the vertical edges of the screen. App developers may want to paint window decorations on there. However, for widgets with gesture detectors, they may want to simply avoid the system gesture recognizer using the gesture inset.

Android Q also provides an API to override some of their system gesture insets if they needed to do so (I haven't gotten around to exposing this API yet). In this case, if both system view and system gesture insets are added together, it's inconvenient for the app developer if they'd like to add a decoration/gesture detector in the area where the system gesture was removed, since they can no longer distinguish between system view insets and system gesture insets.

chinmaygarde

lgtm with nit about the names of the variables.

chinmaygarde · 2019-08-01T23:26:15Z

lib/ui/window/viewport_metrics.h

@@ -59,6 +63,10 @@ struct ViewportMetrics {
  double physical_view_inset_left = 0;
  double physical_view_inset_front = kUnsetDepth;
  double physical_view_inset_back = kUnsetDepth;
+  double system_gesture_inset_top = 0;


Keeping with the naming convention used above, these would be physical_system_gesture_inset_* or physical_gesture_inset_*. I think we should stick to that scheme for consistency. It is also clearer about the fact that those coordinates do not take into account the device pixel ratio.

HansMuller

LGTM

HansMuller · 2019-08-01T23:41:51Z

lib/ui/window.dart

@@ -666,6 +666,24 @@ class Window {
  WindowPadding get viewPadding => _viewPadding;
  WindowPadding _viewPadding = WindowPadding.zero;

+  /// The number of physical pixels on each side of the display rectangle into
+  /// which the application can render, but over which the operating system
+  /// will likely place system gestures.


but over which the operating system will likely place system gestures => but where the operating system will consume input gestures for the sake of system navigation.

HansMuller · 2019-08-01T23:42:16Z

lib/ui/window.dart

+  /// screen, where swiping inwards from the edges takes users backward
+  /// through the history of screens they previously visited.
+  ///
+  /// When this changes, [onMetricsChanged] is called.


this => this property

dnfield · 2019-08-06T20:17:10Z

DEPS

@@ -459,7 +459,7 @@ deps = {
     'packages': [
       {
        'package': 'flutter/android/sdk/platforms',
-        'version': 'version:28r6'
+        'version': 'version:29r1'


How sure are we that this is safe? This means that now all our builds will go against the 29 SDK. Is it stable enough yet?

As mentioned in the buildroot PR (posting here for posterity as well), this isn't certain, but it'd be ideal to support system gestural navigation as soon as possible so that Flutter devs can catch up to this update. Originally, I was using reflection to handle this particular case, but a lint recommended against doing so.

On one hand, we could move forward with using reflection, and once v29 of the Android SDK is stable, roll v29 into the engine and then remove the use of reflection.

Ok. Can we roll th eother SDK packages with this one? Especially build tools.

dnfield · 2019-08-06T20:18:16Z

@matthew-carroll - can you take a look at this from the Android embedding perspective? It looks like this only updates the old embedding but not the new.

… available

…tures

matthew-carroll · 2019-08-06T22:58:56Z

lib/ui/window.dart

-  /// will likely place system UI, such as the keyboard, that fully obscures
-  /// any content.
+  /// which the application can render, but where the operating system will
+  /// consume input gestures for the sake of system navigation.


It's not clear to me based on this description what the difference is between viewInsets and systemGestureInsets. I'm seeing nearly identical descriptions.

I accidentally changed the viewInsets API doc to what I was trying to update systemGestureInsets to. It should be clearer once I've removed this change.

matthew-carroll · 2019-08-06T23:00:46Z

lib/ui/window/viewport_metrics.h

@@ -63,6 +67,10 @@ struct ViewportMetrics {
  double physical_view_inset_left = 0;
  double physical_view_inset_front = kUnsetDepth;
  double physical_view_inset_back = kUnsetDepth;
+  double physical_system_gesture_inset_top = 0;


Are there any implications of defining gesture insets in such a central location? I'm guessing that this definition of viewport metrics is used for all platforms, right? If so, would gesture insets ever make sense on Windows, Linux, Mac, and other non-touch surfaces?

The main reason I decided to define gesture insets this way is that I could foresee other platforms using system gesture navigations as well. I'll need to see how iOS currently handles this, but I know that swiping up from the bottom of the screen on later iPhones is analogous to pressing the home button on older models of the phone.

matthew-carroll · 2019-08-06T23:02:03Z

shell/platform/android/io/flutter/view/FlutterView.java

@@ -100,6 +100,10 @@
        int physicalViewInsetRight = 0;
        int physicalViewInsetBottom = 0;
        int physicalViewInsetLeft = 0;
+        int systemGestureInsetTop = 0;


All changes to this old FlutterView probably need to be made to the new FlutterView, too. Please see io/flutter/embedding/android/FlutterView.java

Also, it would probably be a good idea to introduce a few FlutterView tests around this definition using robolectric.

I started exploring the use of roboelectric, but I'm not very familiar with Flutter's android embedding. Do you have any pointers to a high-level overview of how the embedding works and how to get started writing the test suite for a FlutterViewTest.java?

shihaohong · 2019-08-13T21:06:26Z

Just a heads up that we're currently working on implementing Java tests and the required changes to our unit testing framework, Robolectric, before we can proceed with this PR. See flutter/flutter#38471 for more information.

shihaohong · 2019-08-15T18:02:40Z

After discussion on the state of Robolectric unit testing with Android Q, we've decided to proceed with landing this PR without tests with the intent of adding them once support for Android Q Robolectric testing has been fixed.

I've created a separate issue (flutter/flutter#38626) to track introducing these tests once Robolectric has been upgraded to properly test Android Q.

The PR should be ready for additional review.

cc/ @mklim @matthew-carroll @HansMuller @cbracken

matthew-carroll · 2019-08-15T20:57:52Z

lib/ui/window.dart

@@ -680,6 +680,24 @@ class Window {
  WindowPadding get viewPadding => _viewPadding;
  WindowPadding _viewPadding = WindowPadding.zero;

+  /// The number of physical pixels on each side of the display rectangle into


Not necessary for this PR, but can we follow up with a diagram of the various paddings and insets and put it on the wiki, or on the website, and then maybe link to it from here?

I created an issue (flutter/flutter#38649). some work already exists in the MediaQueryData API docs, we could elaborate on the new insets there and add links to this particular resource once everything has been solidified.

matthew-carroll · 2019-08-15T21:00:19Z

shell/platform/android/io/flutter/embedding/android/FlutterView.java

@@ -312,11 +313,21 @@ public final WindowInsets onApplyWindowInsets(@NonNull WindowInsets insets) {
    viewportMetrics.viewInsetBottom = insets.getSystemWindowInsetBottom();
    viewportMetrics.viewInsetLeft = 0;

+    if (Build.VERSION.SDK_INT >= 29) {


Please use the named constant rather than 29. There's another property within Build that provides it. You should be able to find an example somewhere else in the Android embedding if you look.

I remember that this caused an issue with the pre-submit checks for a reason I did not really understand. I updated it to use the named constant and will let you know if the problem resurfaces.

This has been resolved by suppressing the lint and adding a comment for how the code properly guards for the right SDK version.

matthew-carroll · 2019-08-15T21:01:10Z

shell/platform/android/io/flutter/view/FlutterView.java

@@ -574,6 +579,14 @@ public final WindowInsets onApplyWindowInsets(WindowInsets insets) {
        mMetrics.physicalViewInsetBottom =
            navigationBarHidden ? calculateBottomKeyboardInset(insets) : insets.getSystemWindowInsetBottom();
        mMetrics.physicalViewInsetLeft = 0;
+
+        if (Build.VERSION.SDK_INT >= 29) {


Same comment here about the number 29

matthew-carroll · 2019-08-15T21:01:41Z

shell/platform/android/test/io/flutter/FlutterTestSuite.java

@@ -4,6 +4,10 @@

 package io.flutter;

+import io.flutter.SmokeTest;


If we're not contributing any tests then we should probably revert this file.

…o android-q-gestures

mklim

LGTM

mklim · 2019-08-16T16:25:17Z

shell/platform/android/io/flutter/embedding/android/FlutterView.java

@@ -292,10 +294,14 @@ protected void onSizeChanged(int width, int height, int oldWidth, int oldHeight)
   *
   * This callback is not present in API < 20, which means lower API devices will see
   * the wider than expected padding when the status and navigation bars are hidden.
+   * The annotations to suppress "InlinedApi" and "NewApi" lints prevent lint warnings


really minor nit: This is more of an implementation detail and not really documentation to callers of this method about what it should do. It makes more sense to move it to around the @SuppressLint annotation and comment with // instead of Javadoc (/** */).

mklim · 2019-08-16T16:26:37Z

shell/platform/android/io/flutter/embedding/android/FlutterView.java

@@ -312,11 +318,21 @@ public final WindowInsets onApplyWindowInsets(@NonNull WindowInsets insets) {
    viewportMetrics.viewInsetBottom = insets.getSystemWindowInsetBottom();
    viewportMetrics.viewInsetLeft = 0;

+    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) {


Do you think it be worth it to still add a test for the non Q case in this PR where these are set to zero? It would just basically be verifying the current behavior and making sure that these lines were guarded and didn't cause problems on lower SDK versions. So some but not a huge amount of value from them.

I don't know the root cause of the problem, but I believe that because the TargetApi is set to 20 and the tests only use up to SDK 16 at runtime, the onApplyWindowInsets method gets completely ignored by the test framework (I tried putting print statements in the method and they weren't being called). So, even this simple case might be blocked on upgrading Robolectric :\

Ah, right. The entire method here requires at least API 20, and our setup only includes API 16 right now.

If you wanted to test just the non-Q parts of this method, you should be able to bring in API 21 without any trouble. Robolectric has supported it since 3.0 (we're on 3.8). It's 29/Q specifically that required upgrading to 4.3 and is currently blocked. But doing this has some extra overhead where it would require bringing in another robolectric android package for the right SDK (like robolectric.android-all:5.0.0_r2-robolectric-0) and adding it to CIPD as described here. So at that point I'm not sure if it's worth the hassle. There may be some unknown unknowns that cause it to take extra time.

mklim · 2019-08-16T16:30:29Z

shell/platform/android/io/flutter/view/FlutterView.java

-                mMetrics.physicalHeight, mMetrics.physicalPaddingTop, mMetrics.physicalPaddingRight,
-                mMetrics.physicalPaddingBottom, mMetrics.physicalPaddingLeft, mMetrics.physicalViewInsetTop,
-                mMetrics.physicalViewInsetRight, mMetrics.physicalViewInsetBottom, mMetrics.physicalViewInsetLeft);
+        mNativeView.getFlutterJNI().setViewportMetrics(


This is an aside that's out of scope for this PR, but reading this all of these methods that take 20 identical primitives in a very specific order and then handle them in subtle ways seem a little risky. It would be super trivial to accidentally get the order wrong in one of the many places these are called. In Dart this can be sort of addressed trivially with named params, but in Java/C++ there aren't really great options. Could define a struct/class that has these as named members and then pass that around, but that's probably overkill.

Right, this crossed my mind while making this change. I had at times passed in the wrong number of params and had to rebuild because there were so many of them and they were all identical. Coming up with a cleaner way to handle this will probably help, especially if we see a world where the number of params continue to grow.

… annotation

…engine#10413)

git@github.com:flutter/engine.git/compare/e5f9132b347c...4d5c38e git log e5f9132..4d5c38e --no-merges --oneline 2019-08-16 bkonyi@google.com Roll src/third_party/dart a3b579d5c3..2a3b844b41 (5 commits) (flutter/engine#11060) 2019-08-16 skia-flutter-autoroll@skia.org Roll src/third_party/skia df54f37a5dc1..237a95fe7b28 (2 commits) (flutter/engine#11058) 2019-08-16 chinmaygarde@google.com Roll buildroot to 5a33d6ab to pickup changes to toolchain version tracking. (flutter/engine#11061) 2019-08-16 jason-simmons@users.noreply.github.com Sort the Skia typefaces in a font style set into a consistent order (flutter/engine#11056) 2019-08-16 50503328+flar@users.noreply.github.com Add ccls config files to .gitignore (flutter/engine#11046) 2019-08-16 shihaohong@google.com Pass Android Q insets.systemGestureInsets to Window (flutter/engine#10413) The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff (stuartmorgan@google.com), and stop the roller if necessary.

shihaohong added 3 commits July 31, 2019 16:04

(WIP) Wired inset values to framework

0c1252d

Not able to properly retrieve systemGestureInsets yet

Retrieve systemGestureInsetsTop using reflection

32a6ee7

Implement systemGestureInsets

59c39b5

shihaohong added the platform-android label Aug 1, 2019

shihaohong requested review from HansMuller and jonahwilliams August 1, 2019 18:38

googlebot added the cla: yes label Aug 1, 2019

shihaohong mentioned this pull request Aug 1, 2019

Add MediaQuery.systemGestureInsets to support Android Q flutter/flutter#37416

Merged

9 tasks

shihaohong added 2 commits August 1, 2019 14:18

Update missing documentation

e908f71

Fix arguments on a failing test

0577e2e

chinmaygarde reviewed Aug 1, 2019

View reviewed changes

HansMuller approved these changes Aug 1, 2019

View reviewed changes

Remove reflection getting system gesture insets

f8c3715

shihaohong mentioned this pull request Aug 6, 2019

Roll to Android SDK 29 flutter/buildroot#291

Merged

Update variable names to be consistent with rest of file

a26d365

shihaohong requested a review from dnfield August 6, 2019 18:29

shihaohong added 3 commits August 6, 2019 11:35

Merge with upstream/master

db78d67

Missing updates to merge request

eef5685

Incorporate API doc updates from code review

ebe1348

dnfield reviewed Aug 6, 2019

View reviewed changes

dnfield requested a review from matthew-carroll August 6, 2019 20:17

shihaohong added 5 commits August 6, 2019 14:32

Remove unnecessary _updateWindowMetrics() call, since tearDown is now…

ab80439

… available

Update build-tools and platform-tools versions in DEPS file

0ce87cc

Update to latest references of android SDK

6a37a63

Merge branch 'master' of github.com:flutter/engine into android-q-ges…

d420f27

…tures

Update shell unit tests, update buildroot commit hash

8d78d2a

matthew-carroll reviewed Aug 6, 2019

View reviewed changes

shihaohong added 4 commits August 9, 2019 16:06

(WIP) - Mocked engine for FlutterView unit tests

e560333

Merge with upstream/master

0a465b7

Comment tests back into run_tests.py

ff27613

Fix ordering of android/BUILD.gn sources

bc72f98

shihaohong added the Work in progress (WIP) Not ready (yet) for review! label Aug 12, 2019

shihaohong changed the title ~~Pass Android Q Gestural Navigation Insets~~ Pass Android Q insets.systemGestureInsets to Window Aug 15, 2019

Remove FlutterView tests

98cd1f2

shihaohong mentioned this pull request Aug 15, 2019

Add FlutterView systemGestureInsets tests in the Flutter engine flutter/flutter#38626

Open

1 task

shihaohong removed the request for review from jonahwilliams August 15, 2019 18:03

shihaohong removed the Work in progress (WIP) Not ready (yet) for review! label Aug 15, 2019

shihaohong requested review from matthew-carroll and dnfield August 15, 2019 18:04

Merge branch 'master' into android-q-gestures

a1f6cbf

matthew-carroll reviewed Aug 15, 2019

View reviewed changes

shihaohong added 5 commits August 15, 2019 14:06

Remove changes to FlutterTestSuite.java

7dac454

Update to use VERSION_CODES instead of int value

3f4af17

Merge branch 'android-q-gestures' of github.com:shihaohong/engine int…

a8b17b4

…o android-q-gestures

Add lint suppression for guarded Android Q API calls

8309143

Fix SuppressLint imports and syntax

7f50a08

shihaohong requested review from matthew-carroll and mklim August 16, 2019 15:56

mklim approved these changes Aug 16, 2019

View reviewed changes

Move SuppressLint comment out of Javadoc block to above @SuppressLint…

ab8f418

… annotation

shihaohong merged commit 014ab76 into flutter:master Aug 16, 2019

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Aug 16, 2019

014ab76 Pass Android Q insets.systemGestureInsets to Window (flutter/…

ee6feca

…engine#10413)

engine-flutter-autoroll mentioned this pull request Aug 16, 2019

Roll engine e5f9132b347c..4d5c38e215e9 (6 commits) flutter/flutter#38725

Merged

HansMuller mentioned this pull request Sep 25, 2019

Planned Material Library Features Jul-Sept 2019 flutter/flutter#35088

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pass Android Q insets.systemGestureInsets to Window #10413

Pass Android Q insets.systemGestureInsets to Window #10413

shihaohong commented Aug 1, 2019 •

edited

chinmaygarde left a comment

shihaohong commented Aug 1, 2019 •

edited

chinmaygarde left a comment

chinmaygarde Aug 1, 2019

HansMuller left a comment

HansMuller Aug 1, 2019

HansMuller Aug 1, 2019

dnfield Aug 6, 2019

shihaohong Aug 6, 2019

dnfield Aug 6, 2019

dnfield commented Aug 6, 2019

matthew-carroll Aug 6, 2019

shihaohong Aug 6, 2019

matthew-carroll Aug 6, 2019

shihaohong Aug 6, 2019

matthew-carroll Aug 6, 2019

shihaohong Aug 8, 2019

shihaohong commented Aug 13, 2019

shihaohong commented Aug 15, 2019

matthew-carroll Aug 15, 2019

shihaohong Aug 15, 2019 •

edited

matthew-carroll Aug 15, 2019

shihaohong Aug 15, 2019

shihaohong Aug 16, 2019

matthew-carroll Aug 15, 2019

matthew-carroll Aug 15, 2019

mklim left a comment

mklim Aug 16, 2019

mklim Aug 16, 2019

shihaohong Aug 16, 2019 •

edited

mklim Aug 16, 2019 •

edited

mklim Aug 16, 2019

shihaohong Aug 16, 2019

		@@ -4,6 +4,10 @@

		package io.flutter;

		import io.flutter.SmokeTest;

Pass Android Q insets.systemGestureInsets to Window #10413

Pass Android Q insets.systemGestureInsets to Window #10413

Conversation

shihaohong commented Aug 1, 2019 • edited

Description:

Issues:

Todos:

chinmaygarde left a comment

Choose a reason for hiding this comment

shihaohong commented Aug 1, 2019 • edited

chinmaygarde left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HansMuller left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dnfield commented Aug 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shihaohong commented Aug 13, 2019

shihaohong commented Aug 15, 2019

Choose a reason for hiding this comment

shihaohong Aug 15, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mklim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shihaohong Aug 16, 2019 • edited

Choose a reason for hiding this comment

mklim Aug 16, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shihaohong commented Aug 1, 2019 •

edited

shihaohong commented Aug 1, 2019 •

edited

shihaohong Aug 15, 2019 •

edited

shihaohong Aug 16, 2019 •

edited

mklim Aug 16, 2019 •

edited