Animation Smoothness Explainer #1003
Conversation
|
I'm genuinely excited to see your proposal regarding the ability to track smoothness in web applications with browser API. In our own product (Excel Online), we would definitely make extensive use of an API like this, and I’m confident many other web applications would benefit from it as well. Over a year ago, I built an internal infrastructure aimed at monitoring animation smoothness, and I’d love to share some insights based on that experience. One key aspect I believe is critical, you did mention in the background section, is the FPS consistency. However, I didn't see it clearly addressed in the specific API options. FPS consistency is crucial because it helps identify the exact points where performance drops occur. For example, you might have a 10-second animation that appears fine in aggregate, but actually experiences complete frame drops for half a second, which wouldn't be reflected in an average FPS calculation over the full duration. I believe the API output should provide enough detail to estimate frame drops more accurately. This could be achieved by either returning the timestamps of received frames, allowing us to calculate the gaps ourselves, or by letting the user specify a time interval (e.g., 100ms) to compute localized FPS values. This is the approach we took internally, and it significantly improved the accuracy of our measurements. Of course, having a native API to rely on, rather than using requestAnimationFrame and its limitations, would be a great advantage. If you'd like, I'd be happy to continue the discussion offline and dive deeper into the topic. |
|
@mmocny did a bunch of work on this. It might be worthwhile for y'all to chat :) |
|
Indeed! We started some chatting at BlinkOn last week, and I've started to review this explainer. Will send comments soon. |
|
@microsoft-github-policy-service agree company="Microsoft" |
| ## Goals | ||
| * Webpage accessible API that captures user-perceived framerate accurately, taking into account both the main and compositor threads. | ||
| * An approach that doesn’t cause webpage performance regressions. | ||
| * Enabling a web developer to control what time interval is considered. |
There was a problem hiding this comment.
In my mind, if an api is intended for a developer to use in real time and actually update the UX, then it might not be a good fit for the performance timeline (which tries to fire observers is a lazy, decoupled fashion).
Perhaps something more akin to Compute Pressure API, or performance.measureUserAgentSpecificMemory(), or navigator.connection.*?
|
|
||
| ### 2. Measuring animation and graphics performance of browsers | ||
|
|
||
| The public benchmark, MotionMark, measures how well different browsers render animation. For each test, MotionMark calculates the most complex animation that the browser can render at certain frame rate. The test starts with a very complex animation that makes the frame rate drop to about half of the expected rate. Then, MotionMark gradually reduces the animation's complexity until the frame rate returns to the expected rate. Through this process, MotionMark can determine the most complex animation that the browser can handle while maintaining an appropriate frame rate and uses this information to give the browser a score. To get an accurate score, it is crucial that MotionMark can measure frame rate precisely. Currently, MotionMark measures frame rate based on rAF calls, which can be impacted by other tasks on the main thread besides animation. It also doesn't take into account animations on the compositor thread. The method using rAF to measure frame rate doesn't reflect the user's actual experience. |
There was a problem hiding this comment.
Agree with these points, but I don't think we need a public web exposed api to address the benchmarking use case.
|
|
||
| Animation frames are rendered on the screen when there is a change that needs to be updated. If they are not updated in a certain amount of time, the browser drops a frame, which may affect animation smoothness. | ||
|
|
||
| The rAF method has the browser call a function (rAF) to update the animation before the screen refreshes. By counting how often rAF is called, you can determine the FPS. If the browser skips calling rAF, it means a frame was dropped. This method helps understand how well the browser handles animations and whether any frames are being dropped. |
There was a problem hiding this comment.
Nit: "it means a frame was dropped" is somewhat of an over-loaded term.
I think "a potential frame rendering opportunity was skipped" might be more appropriate.
I only differentiate because I think there is a difference between an animation frame which the user agent wanted to present not being able to get presented within acceptable latency, vs the user agent choosing to throttle frame rendering opportunities to balance tradeoffs.
A web developer measuring using raf loops is not really able to differentiate.
| The rAF method has the browser call a function (rAF) to update the animation before the screen refreshes. By counting how often rAF is called, you can determine the FPS. If the browser skips calling rAF, it means a frame was dropped. This method helps understand how well the browser handles animations and whether any frames are being dropped. | ||
|
|
||
| #### Limitations | ||
| Using rAF to determine the FPS can be energy intensive and inaccurate. This approach can negatively impact battery life by preventing the skipping of unnecessary steps in the rendering pipeline. While this is not usually the case, using rAF inefficiently can lead to dropped or partially presented frames, making the animation less smooth. It’s not the best method for understanding animation smoothness because it does not take into account factors like compositor offload and offscreen canvas. While rAF can be useful, it isn’t the most accurate and relying on it too heavily can lead to energy waste and suboptimal performance. |
There was a problem hiding this comment.
(The point about compositor frames is valid, but I want to focus on rAF-polling for a second).
In my experience, it's not just the scheduling of rAF task itself that significantly affects performance, but rather the downstream effects of raf-polling on scheduler decision making abilities, can lead to janky moments.
For example, when I keep calling requestAnimationFrame() at 60hz, Chromium will schedule a BeginMainFrame task at the start of each new vsync. No other rendering opportunity will be given to the page until the next vsync.
If your rAF() call does nothing but measure timings, and doesn't update UI, now your page effectively becomes "static" for the next 16ms.
And that means that all "real" UI updates to the page, which are scheduled in the middle of a frame-- such as interactions-- now won't be able to paint until after the next vsync.
In theory, that only delays things a few ms, but in practice what happens is:
- rAF()
- Important UI update (perhaps new UI Event).
- Runs quickly, much less than 16ms.
- Requires a rendering update
- idle time, waiting for next vsync
- ...because of the rAF() at the start of the vsync.
- If not for that, we would be able to schedule a render opportunity nearly ~immediately.
- Idle time allows us to schedule other work, perhaps even idle callbacks...
- potentially long running
- Vsync fires and we schedule BMF, but now main thread is blocked and busy...
- potentially for more than 16ms and we start to "drop" frames.
Chromium is currently experimenting with deferring task scheduling under certain cases (i.e. immediately after interaction UI Event dispatch) in order keep scheduler idle for a little while until next rendering opportunity. But now we are reducing throughput in order to optimize latency. And this new scheduling policy doesn't help with more general case.
Also, nit: Cannot you use rAF for OffscreenCanvas measurement (via the worker itself)? Did you mean desynchronized canvas?
| ### Long Animation Frames API | ||
|
|
||
| #### Description | ||
| A long animation frame (LoAF) occurs when a frame takes more than 50ms to render. The Long Animation Frames API allows developers to identify long animation frames by keeping track of the time it takes for frames to complete. If the frame exceeds the threshold, it is flagged. |
There was a problem hiding this comment.
I would add that one key difference is that rAF() is an explicit request for scheduling a rendering opportunity, paired up with a callback (an "event") to observe that rendering opportunity.
LoAF is just a means for observing rendering, it doesn't affect scheduling. I think this makes it a better fit.
(The threshold values can perhaps be easily adjusted for this use case?)
| ### FPS-Emitter | ||
|
|
||
| #### Description | ||
| In the past, Edge had a library called fps-emitter that emits an update event. Once a new instance of FPS-emitter is called, it starts tracking frames per second via the rAF method. When the FPS changes, the result is returned as an EventEmitter. This method builds off the rAF method described above and faces similar limitations. |
There was a problem hiding this comment.
I've built something like this before using rAF() on main thread, and then also rAF() on OffscreenCanvas, and coordinating between the two.
I could then compare in real time when main is lagging / throttled / dropping frames.
I still think this comes with all the baggage of rAF, but have you played with that approach as well?
| `window.addEventListener("frameratechange", (event) =>{doSomething();})` | ||
|
|
||
| ## Concerns/Open Questions | ||
| 1. The user-perceived smoothness is influenced by both the main thread and the compositor thread. Accurate measurement of frame rates must account for both. Since the compositor thread operates independently of the main thread, it can be difficult to get its frame rate data. However, an accurate frame rate measurements needs to take into account both measurements. |
There was a problem hiding this comment.
I think that this is true for a benchmark / internal browser measurement metric. But for a web exposed API where developers are expected to act and adjust content to fit within constraints, I'm not sure if we really need more than main thread frame rendering signals... If the compositor is struggling to keep up with frame rates, I think it will put back pressure on the main thread scheduling opportunities, anyway.
If you have compositor-driven-animations, then developers typically expected these to be fast and efficient, and not typically able to "adjust down" the quality. It's theoretically possible, but I'm not sure it would actually happen.
Net/net, I think I would focus on:
- A good main-thread "smoothness" metric/signal for web developers to use
- A good overall "smoothness" metric for browser venders to use
Its really only for competitive benchmarking purposes that you sort of want both available in a browser, but perhaps you can just use Worker rAF()?
| 1. The user-perceived smoothness is influenced by both the main thread and the compositor thread. Accurate measurement of frame rates must account for both. Since the compositor thread operates independently of the main thread, it can be difficult to get its frame rate data. However, an accurate frame rate measurements needs to take into account both measurements. | ||
| 2. Similar to the abandoned [Frame Timing interface](https://wicg.github.io/frame-timing/#introduction). We are currently gathering historical context on how this relates and why it is no longer being pursued. | ||
| 3. Questions to Consider: | ||
| * Should content missing from the compositor frame due to delayed tile rasterization be tracked? |
There was a problem hiding this comment.
Some folks here have started calling this concept frame "contentfulness", to differentiate raw frame "smoothness".
It can include more than just tile raster (such as image decode), including the the actual design of page contents (such as video bitrate / buffering, or real time video game latency, or typical web page loading related layout shifts, images, ads, fonts...). Users seem more sensitive to that stuff than literal graphics pipeline performance (if trying to judge "quality of experience").
|
Merging initial draft to get a static URL that we can share. Will address outstanding feedback in next iterations. |
No description provided.