Performance benchmarks

[Almouro]

Hi, thanks for creating this space! ❤️

I'm currently running some performance benchmarks comparing old arch and new arch and the results I'm getting are surprising to me.

I'm using

• a low end Samsung J3 2017 Android 9 (on armeabi-v7a) for the videos
• a low end Samsung A10s Android 10 (on armeabi-v7a) on AWS device farm for the measures

Scenario 1: comparing rendering 10k views

WINNER New arch ✅

The winner is clearly the new arch here and by far. Kudos on that! 🤩

2022-10-18.13-19-38.mp4

I'm also running some perf metrics comparison using our own tool (https://github.com/bamlab/android-performance-profiler) that's basically a glorified adb shell top

The full report here makes it clear that this is because the new renderer is much more efficient than the old bridge

Scenario 2: comparing rendering 2k texts

WINNER Old arch ❌

2022-10-19.14-22-08.mp4

This time the old arch is faster, mainly because of heavier UI thread consumption

All the code is here.

This might have a big impact on performance of apps migration:

• I can reproduce similar results in a 3rd scenario, displaying some Tweets components
• This might be related to performance issues mentioned on the Flashlist migration to Fabric New architecture support Shopify/flash-list#550

I haven't tried to go deeper with a system trace to check what was going on yet, and my understanding of how the new arch works and implements layouting is limited still.

But I'm hoping to get a discussion started towards improving performance (and/or the benchmarks) and what could be done, if someone already has some ideas ☺️

[psionic12]

Same here, I tested the rn-tester and the new arch is about 50ms slower than the old arch, after testing piece by piece, it seems that the layout procedure in new arch takes more time. I'm not sure if it is because of the performance of the code in UIManger, or the new arch somehow make the view hierarchy more complicated than before.

[sammy-SC]

Hello Almouro,

First of all, thank you very much for sharing your benchmarks, as well as a playground where we could look into and understand what is happening.

I’ve personally tried to reproduce your results on both a Pixel 5 and a Pixel 3 physical device. I ran many-texts and 100-tweets scenarios. On both devices the CPU performances were about neutral. We’re aware of the higher memory consumption. The new architecture has complete in memory representation of UI. This adds the extra memory cost but it unlocks new features like concurrent rendering. We’re looking into ways to optimise memory usage.

To better understand what would be the culprit here, I’d like to ask follow up questions:

1. Have you tried running the benchmarks on more devices?
2. Is there a specific reason why you used a device farm (AWS) for measurements?
3. Have you tried comparing performance on iOS?

I’d like to take the opportunity to say that the React Native team is fully committed to rollout the New Architecture of React Native, without introducing regressions for our users. I’m personally interested in taking a closer look at any well-reported regression.

To give you a bit of context, performance was a big concern during the rollout of the New Architecture in our internal products. Our internal performance evaluation was neutral across all React Native surfaces in the Facebook app on both Android and iOS. In the last years, we conducted dozens of tests in production on millions of devices to assure performance was neutral. Still, we’re open to looking into benchmarks like the one you reported to understand if there are regression we might have missed.

Sam on behalf of the React Team.

[Almouro]

Hi @sammy-SC! Thanks for your answer, and for providing the measures! 🙏

Apologies for the long message 😅
(btw I'm also available on the RN contributors discord to discuss further if you prefer it!)

• Quick question first, what were the devices for which the Facebook app performance impact was tested? (or was it data from production users?)

Checking your results

Worth noting here that the Pixel 3 and Pixel 5 are quite high-end, so if there is in fact a performance impact, it might be less noticeable.
Even so, if we reduce the measures you provided to a duration of 3s since both versions finish in ~3s on your device, this is the result.
They're quite similar, but I'm still seeing the increased TTI and a 27% overhead in UI thread CPU consumption.

New measures with high end device

My hypothesis is that with more tweets you'd see the impact on the high-end phone
If you want to try it out, I've added

• APKs with 300 tweets (here and here)
• APKs with 10k texts (here and here)

I've tried those out on my Samsung S10 (even higher end):

• 300 tweets
  • impact seems low, still TTI is up ~10%, UI thread up 12%
• 10k texts
  • again we see like below the big impact for text rendering, mainly coming from heavier thread usage
    

Choice of test devices

Using Geekbench CPU benchmarks for example, here is a comparison of CPU performance for all the devices used in this conversation so far:
(I'm also adding my own S10 on which I ran some new measures)

Device	Pixel 5	Pixel 3	J3	A10s	S10
Single-core score	570	448	104	135	681
Multi-core score	1533	1629	354	480	2452

The reason for using such low end devices for testing is that we found recently that a lot of our users had quite low end devices, but I imagine the FB app would be quite similar.

Based on that small research, roughly 25% of our Android users would basically have a worse phone than the A10s I'm testing on.
This is not even taking into account our Android TV projects which have to face even crappier devices 😅

About Flatlist

I might have missed something here, but Flatlist seems to be consistently performing worse on low end and high end Android phones. Even on my S10, scroll locks which seems to correlate with heavy fabric_bg usage report and code + APKs

It's true that on the old arch, Flatlist is not the best component anymore, and I think internally FB uses something else?

But the thing is the new Flashlist is struggling with performance with the new architecture, seemingly for similar reasons Shopify/flash-list#550 (comment)

On the issue, @j-piasecki shares a system trace on which we see updateState for texts being perhaps the bottleneck

My current conclusions and concerns

• Of course, no apps should be rendering thousands of components. I also understand performance gains should not necessarily be expected especially for such simple scenarios (I'm not even taking advantage of concurrent rendering anywhere)

• But in most scenario closer to real life (for instance this one, https://github.com/Almouro/fabric-test#scenario-6-navigation), the new arch seems to be consistently performing a bit worse. Like 10% increased CPU usage and render time, which is not negligible.
  On high end devices, the impact seems less clear indeed.
  Mainly the impact seems to be on the UI thread, which is concerning to me, since this might mean low end devices struggle more to achieve 60FPS

• Might be wishful thinking, but maybe there's just something to be improved for Text or layouting?
  When I see the fantastic improvement we have for simple views I do wish we'd have something similar for texts 🤞

[sammy-SC]

I'll answer in several parts to break up the discussion into pieces. I'm happy to talk on Discord as well but I feel like it is more accessible/searchable on Github.

Quick question first, what were the devices for which the Facebook app performance impact was tested? (or was it data from production users?)

It was tested with data from real world users. Surfaces in Facebook app are instrumented so we can measure how long it takes from user initiating navigation until a marker element is rendered on screen. Collected data was grouped by device class: low end, mid end and high end to make sure all types of devices have neutral perf.

[sammy-SC]

Pixel 3 was the oldest found I could get my hands on the day I was re-running your benchmarks. Low end Android devices are important. As I mentioned in my previous comment, we were breaking down perf metrics by device class to make ensure perf neutrality for all device classes. The new architecture should not perform worse on low end devices than the old architecture.

Where in the results do you see TTI? is it "Processes with high CPU usage detected"?
Why do you think higher CPU UI thread utilisation is bad? The metric is about how much CPU is waiting for cache or spending time context switching. it is worth keeping track of it but I don't understand how it makes the experience worse for the end user.

[sammy-SC]

Of course, no apps should be rendering thousands of components. I also understand performance gains should not necessarily be expected especially for such simple scenarios (I'm not even taking advantage of concurrent rendering anywhere)

Even though no app will render thousands of components, I think this is a good exercise. When I was sharing details about how we did rollout, the point was that we were looking at real world usage of the framework. Examples likes this can point to perf bugs, which is something I hope we can discover and fix.

Might be wishful thinking, but maybe there's just something to be improved for Text or layouting?

This certainly can be the case. The new architecture uses lower level Yoga APIs. This means more responsibility is on side of RN than before. Previously, the Yoga tree was constructed through high level interface and everything was taken care of. There is a trade off between how much RN has control over things and simplicity of Yoga integration.
Also the way text is measured has changed. The critical path is in C++ and it could be that we do unnecessary copy of heavy objects or text measurements are not cached correctly.
I'll run the new benchmarks you prepared (thank you!) and see if I can get a consistent repro. When I have a repro I can dig in and see look at what's going on.

[Almouro]

Hi @sammy-SC, thanks for your answers! I really appreciate you going out of your way to reply 🙏

TTI

in this report, I was referring to the TTI from the average run time, since the test just waits for element to appear.
But the measure isn't accurate, so it might not reflect the reality -> it's just measuring the Appium test execution and I've just realized that Appium waits for UI thread to be done to search for elements, so the measure might just be wrongly higher just because of the higher UI thread usage

But to get my TTI results, I was just manually timing the appearance of elements by comparing videos (like this video for 100 tweets on low end devices. At least on low end devices, TTI was consistently ~20% worse.

UI Thread

• Why do you think higher CPU UI thread utilisation is bad?

Since the UI thread is still ultimately responsible for orchestrating the frame rendering at 60FPS (or more), my understanding is that more pressure on the UI thread might make it harder to achieve perfect frame rate and induce higher risk of lag.
Especially when rendering lots of elements (like in list scrolling), or for users with low end phones with bad CPU cores that might get saturated easily.

To be honest, it's also the correlation I could find with the higher TTI, but that was mainly guessing.
When I'm taking a callstack trace with Android studio for the example with thousands of texts, the higher TTI seems to correlate with a few more seconds spent on the UI thread for ReactTextViewManager.updateState
@psionic12 seems to have found very interesting insights here (#88 (comment)): mainly that this might be linked to too much JNI calls to read from native maps

[sammy-SC]

Hello @Almouro,

we spent some time analysing performance and summarised our findings in #123.

If you have some time, could you please try to run your benchmarks on the 0.72.0-RC.1 and see what the performance is like? We've made some improvements.