-
Notifications
You must be signed in to change notification settings - Fork 6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[iOS] Avoid jitter and laggy when user is dragging on iOS Promotion devices #35592
Conversation
Tested up on my iPhone 13 Pro iOS 15.6 Works like a charm. |
@rotorgames Oh, I’m very happy to see that😄 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for providing the fix.
I tested the issue and found that there were 60hz frames even when doing automated scrolling. (Set a timer to scroll the list view periodically without touch events).
So I suspected it had more to do with gesture handled in 60hz.
This fix seems pretty similar to the workaround in flutter/flutter#101653 (comment), does this fix also have the issue where the first frame when scroll just starts is 60hz? (I can test it when I get a chance too).
shell/platform/darwin/ios/framework/Source/FlutterViewController.mm
Outdated
Show resolved
Hide resolved
@cyanglaz Thx for reviewing this😄. I m not sure first frame is 60HZ or 120hz.But I activate the vsync client when receive first touch event.So I think it should be 120HZ.But I think it should be ok if first frame is 60HZ.Making sure the touch events delivered with 120HZ continuously is the core target of this PR In my opinion, this solution is different from that solution you mentioned(About the In that solution, it will start a ticker and let But for solution in this PR, the touch events delivery rate will be 120HZ, which will solve the root cause and it is expected. You can check this iOS native demo to prove this Demo codeclass HomeViewController: UIViewController {
private lazy var mainDisplayLink = CADisplayLink(target: self, selector: #selector(self.onDisplayLinkInMainThread))
private lazy var uiTaskRunnerDisplayLink = CADisplayLink(target: self, selector: #selector(self.onDisplayLinkInUITaskRunner))
private var oldTime:CFAbsoluteTime = 0
override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {
let current = CACurrentMediaTime()
let intervalMs = current - oldTime
print("interval of touchesMoved method is :\(intervalMs) s")
oldTime = current
}
var thread:Thread!
override func viewDidLoad() {
super.viewDidLoad()
self.view.backgroundColor = .red
// Add a display link in UITaskRunner like what we do in VsyncWaiterIOS.
// If we only run this line , the touch callback interval is 16ms.
self.addDisplayLinkInUITaskRunner()
// You can try to uncomment the line below to see the log.
// The touch callback interval will become 8ms.
// Which is correct and expected on Promotion devices.
// self.addDisplayLinkInMainThread()
}
//====== Add displaylink in MainThread
func addDisplayLinkInMainThread() {
if #available(iOS 15.0, *) {
self.mainDisplayLink.preferredFrameRateRange = .init(minimum: 60, maximum: 120, preferred: 120)
} else {
self.mainDisplayLink.preferredFramesPerSecond = 120
}
self.mainDisplayLink.add(to: .current, forMode: .common)
}
//====== Add displaylink not in main thread,
// We create a new thread to simulate the UITaskRunner in flutter engine
func addDisplayLinkInUITaskRunner() {
thread = Thread(block: {
RunLoop.current.add(NSMachPort(), forMode: .common)
RunLoop.current.run()
})
self.thread.name = "UITaskRunner"
self.thread.start()
self.perform(#selector(addDisplayLink), on: thread, with: nil, waitUntilDone: false)
}
@objc func addDisplayLink() {
if #available(iOS 15.0, *) {
self.uiTaskRunnerDisplayLink.preferredFrameRateRange = .init(minimum: 60, maximum: 120, preferred: 120)
} else {
self.uiTaskRunnerDisplayLink.preferredFramesPerSecond = 120
}
self.uiTaskRunnerDisplayLink.add(to: .current, forMode: .common)
}
@objc func onDisplayLinkInUITaskRunner() {
// print("onDislpayLink on UITaskRunner")
}
@objc func onDisplayLinkInMainThread() {
// print("onDislpayLink on main thread")
}
} |
And I think this PR maybe not related with the automated scrolling, this PR is to make sure the delivery rate of touch events is correct, which is driven by touch events. But automated scrolling is driven by animation system...So I think that should be different from this. |
Edit: I missed the part where the vsync waiter in rotorgames@37b78ae was changed to be on main thread. So it is a similar solution but creating a new vsync waiter on main thread. This seems better as we can keep the UIThread vsync waiter unchanged. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tested locally as well it works nicely!
shell/platform/darwin/ios/framework/Source/FlutterViewController.mm
Outdated
Show resolved
Hide resolved
shell/platform/darwin/ios/framework/Source/FlutterViewController.mm
Outdated
Show resolved
Hide resolved
shell/platform/darwin/ios/framework/Source/FlutterViewController.mm
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@@ -65,6 +65,9 @@ @interface FlutterViewController () <FlutterBinaryMessenger, UIScrollViewDelegat | |||
@property(nonatomic, assign) double targetViewInsetBottom; | |||
@property(nonatomic, retain) VSyncClient* keyboardAnimationVSyncClient; | |||
|
|||
/// VSyncClient for touch callback's rate correction. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nits:
/// VSyncClient for touch callback's rate correction. | |
/// VSyncClient for touch callback's frame rate correction. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe also add a short explanation about why this is needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@@ -671,6 +674,9 @@ - (void)viewDidLoad { | |||
// Register internal plugins. | |||
[self addInternalPlugins]; | |||
|
|||
// Create a vsync client to correct touch rate if needed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nits:
how about: Create a vsync client to correct frame rate during touch events if needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have changed it to delivery frame rate of touch events
, which seems to be more accurate.😄
@@ -966,6 +973,9 @@ - (void)dispatchTouches:(NSSet*)touches | |||
} | |||
} | |||
|
|||
// Activate or pause touch rate correction according to the touches when user is interacting. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"touch rate" is a little ambiguous. maybe "frame rate during touch events"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have changed it to delivery frame rate of touch events
, which seems to be more accurate.😄
@iskakaushik Could you do a secondary review? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering if this PR is maybe a bit of overkill? Similar effect can be achieved much easier. Look at knopp@50178b7.
We can just delay pausing the CADisplayLink for few frames. That way it is continuously enabled while user is interacting with the application and then disabled when application gets idle.
Hi @knopp |
@knopp |
@iskakaushik Would you mind taking a look^_^ |
There are some usage of "junk" in the code comments as well, could you also fix them? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Blocking landing until the "junk" usages are removed in comments.
@cyanglaz |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
auto label is removed for flutter/engine, pr: 35592, due to - The status or check suite Linux Framework Smoke Tests has failed. Please fix the issues identified (or deflake) before re-applying this label. |
Unfortunately I think you're hitting something relating to #35843, can you rebase onto top of tree? |
Done^_^ |
hello, our team is wondering which release of flutter this fix will be part of? Is it part of stable release 3.3.2 already? or will it be part of 3.3.7? thank you for your help! |
This is available in 3.4.0-27.0.pre. |
thank you @jmagman 💯 |
hello everyone and @jmagman do you know if this is part of stable release 3.7.0 (macos) that got released on 1/24/23? thank you very much! |
Yes it is on stable. |
thank you so much @cyanglaz ! appreciate it! This is awesome news! we will try 3.7.0 then soon. |
Related issue to fix:
Fixes flutter/flutter#101653
Behavior
When user is dragging (means the finger is on screen and move continuously) on the screen on Promotion devices on iOS(iPhone13Pro or iPhone13ProMax), we will see a very transparent frame drop and junk behavior.
But when user is dragging on normal devices without Promotion (iPhone12 or iPhone11..), the drag behavior seems to be very smooth.
Root Cause
The touch events delivered by
UIViewController
liketouchesBegan
,touchesMoved
is not called with 120HZ rate. Normally, it is called with 60HZ rate, which will cause that the signal's rate delivered by platfrom doesn't match the rate using in render pipeline inVsyncWaiterIOS
. So it will have a gap, which will lead junk and laggy. This mechanism is the same as flutter/flutter#109435 .Why
For iOS promotion devices, system detects that there is no 120HZ animation running, so it calls touch events with 60HZ rate (maybe for saving more power). We already have a 120HZ
CADisplayLink
inVsyncWaiterIOS
, but unfortunately it is run inUITaskRunner
, which is not iOS main thread.(On iOS, all native UI animations should run in main thread, so system doesn't detect that and let gesture system remain 60HZ callback rate)Here is the iOS native demo code to show this mechanism.
You can run it on Promotion device to prove this.
Demo code
Solution
We can't simply move the
CADisplayLink
inVsyncWaiterIOS
fromUITaskRunner
toPlatformTaskRunner
(or just move itsCADisplayLink
tomainRunloop
), because flutter UI tasks should be inUITaskRunner
.So, we can create a new
VsyncClient
inFlutterViewController
.And add it in
mainRunloop
. When user is interacting, we should activate it to tell system: "I have 120HZ animation running, plz give me the touch callback with 120HZ rate !! "And we pause it when user ends interaction.
But we only do this when it is needed indeed.
For exmaple, if a device's max frame rate is 60HZ, means we don't need to create new
VsyncClient
to correct the touch rate.Moreover, in the callback of this vsync client, we do nothing, because this
VsyncClient
is to trigger system to callback touch events with 120HZ rate. So I think this cost is acceptable.Test locally to see the comparsion with this fix
Flutter demo code
The recorded screen video can't show this fix very transparently (because the video is 45FPS - 55FPS). So I very recommend you to test this fix locally with physical devices.
Flutter framework ref can run this locally:
d0c8774508808a0cc56d6c1f07c4200f380a480c
Flutter engine ref: This branch
Device: iPhone13 Pro or iPhone13 Pro Max
You can just run the code to see the behavior with this fix
or delete the method
createTouchRateCorrectionVSyncClientIfNeeded
to see the behavior beforePre-launch Checklist
writing and running engine tests.
///
).