-
Notifications
You must be signed in to change notification settings - Fork 27.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pointer event resampler (#41118) #60558
Pointer event resampler (#41118) #60558
Conversation
f67f226
to
feecc91
Compare
2113a47
to
c29a076
Compare
PTAL |
nit: Can you please add a short description to the PR describing what this does and what it is for? Also, are there any issues associated with this? If so, please link them from the PR description. |
Done. Sorry for the lack of explanation earlier. We're planning to have this land together with a potential blog post that explains it in more detail but please let me know what more documentation would be useful in code comments and PR description as reading the post should not be necessary to understand this. |
packages/flutter/test/gestures/pointer_data_resampler_test.dart
Outdated
Show resolved
Hide resolved
8e967bf
to
41db16a
Compare
I don't mean any review comment or suggestion for modification, but I want to point out the thing I wrote in #60796 (comment) . |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The algorithm and the unit tests in this PR looks good. Here are a few more things to consider in my opinion in order to make this out of draft and ready to merge:
-
Use this in
GestureBinding
and expose a boolean flag to turn this resampling on or off. In the beginning, we'll make it default to off and only enable it in testing. Once we're confident with those tests, turn it to on by default. The use of resampling probably would happen insideGestureBinding._handlePointerDataPacket
wherePointerData
is converted toPointerEvent
and added to the queue. -
Either in
GestureBinding
orPointerDataResampler
, document why someone would want to resample events, and why someone would not want to resample events. -
Decide whether we want to resample at
PointerData
level orPointerEvent
level. According to previous discussion, as long as we're only exposing this resampling as a global flag instead of a widget-level flag, this shouldn't matter much. The small advantage ofPointerEvent
is a closer integration withflutter_test
package, and it's possible for us to enable widget-level resampling in the future. The small advantage ofPointerData
is that it's closer to the platform implementation (e.g., Fuchsia), it has a simpler class inheritance structure (no subclass such asPointerAddEvent
,PointerRemoveEvent
, ...), and it may allow us to test lower level part of the system that's inaccessible toPointerEvent
tests.
@dreveman told us this Monday that he's working on an updated resampler for |
41db16a
to
c8816c1
Compare
9412299
to
2d8352d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
a2ae63d
to
d0ecda2
Compare
I tried this flag on https://github.com/flutter/flutter/tree/master/dev/benchmarks/complex_layout
It failed and this is what I got:
Did I misunderstood how it should be used? |
What platform is this? Looks like either currentSystemFrameTimeStamp is not set currently or event timestamps are not set correctly. |
I tested it on a Pixel 4, with |
FYI, if I add
before that
which looks kind of reasonable numbers but for the time difference. If I wait longer time to touch the screen after the app starts, I get larger time difference. It seems that |
If I also add
I guess the timestamp is not yet initialized to the value we expect. According to https://api.flutter.dev/flutter/scheduler/SchedulerBinding/currentFrameTimeStamp.html I believe that's because we are not using the timestamp within a frame scheduling process. And Comment out this |
This resampling logic depends on currentSystemFrameTimeStamp being a valid time stamp for the expected presentation of the next frame. Hence, always in the future. That's what it is on Fuchsia. We can adjust this logic to expect something else on a different platform but ideally we'd instead make currentSystemFrameTimeStamp the same across all platforms. |
6a63be1
to
c110ad6
Compare
This should work fine now. I found a bug in the copyWith logic that caused down/move/up events to not be produced correctly. Added some tests for this. Latest version of this change also contains a debug flag that is used to determine if resampling works correctly on a device. I've verified that it works correctly on a Pixel 3 device. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I wrote a unit test (or in this context, more close to an integration test?) for the resampling performing on widgets, by tracking the events from Listener
, which should be a natural unit test for the sampling working with widgets. I'll file another PR for that after this lands and hope you can review that.
c110ad6
to
8c71069
Compare
This introduces the PointerEventResampler class that contains the core logic needed to resample pointer events. This can be used to get smoother event processing at the cos of some added latency. Devices with low frequency sensors or when the frequency is not a multiple of the display frequency (e.g., 120Hz input and 90Hz display) benefit from this. The caller is responsible for deciding what sampling offset to use and what frequency to sample at. If touch events arrive at 65 Hz, then it's typically good to be using a sampling offset that is current time - 22 ms for smooth touch event delivery. This change is also using the PointerEventResampler class to add resampling support to GestureBinding. Resampling is disabled by default. This can be used to fix: flutter#41118
8c71069
to
b7802f6
Compare
|
||
final PointerEventResampler resampler = _resamplers.putIfAbsent( | ||
event.device, | ||
() => PointerEventResampler(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it true that when this constructor is called (i.e., event.device
key is absent in _resamplers
), event
can only be PointerAddedEvent
or PointerDownEvent
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, that would only be true if resampling was enabled before the first pointer event. If you enable resampling after first receiving a down event then we'll need to create the resampler here when processing a move event.
|
||
if (resamplingEnabled) { | ||
_resampler.addOrDispatchAll(_pendingPointerEvents); | ||
_resampler.sample(samplingOffset); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this mean that, when the pending event queue in the resampler is not empty, but there are no more future inputs, these pending events will stay in the queue and will not be processed, since _flushPointerEventQueue is not called?
I realize this when I'm trying to move resampler to handlePointerEvent.
cc @liyuqian
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, the _Resampler
will schedule _handleSampleTimeChanged
after frame callback which will call _flushPointerEventQueue
. See: https://github.com/flutter/flutter/pull/60558/files/b7802f69f91e1b5466d6b5b58e98193160964ca5#diff-6305361e0c7677f0eebfb05dfcbe0336
Pointer event resampler (#41118)
This introduces the PointerEventResampler class that contains
the core logic needed to resample pointer events. This can be used
to get smoother event processing at the cos of some added latency.
Devices with low frequency sensors or when the frequency is not a
multiple of the display frequency (e.g., 120Hz input and 90Hz
display) benefit from this.
The caller is responsible for deciding what sampling offset to use
and what frequency to sample at. If touch events arrive at 65 Hz,
then it's typically good to be using a sampling offset that is
current time - 22 ms for smooth touch event delivery.
This change is also using the PointerEventResampler class to
add resampling support to GestureBinding. Resampling is disabled
by default.
This can be used to fix: #41118