Chrome 79 - 96 results

[krausest]

Results for Chrome 79 are online. Sadly this benchmark reports one again worse numbers for a newer chrome version:

[leeoniya]

wow, is 79 really 40% slower at dom creation than 75 😮 .

might be interesting to go back even further to see where the inflection point was. maybe now that they have huge market share "fastest browser" marketing has greatly diminished ROI?

is every version tested with the same OS-level Spectre and Meltdown mitigations, cause if you're just plotting historical data, then this could easily be a large factor (and worth re-testing v75 again). on top of that, v8/Blink itself might be disabling some JIT paths that lead to possible timing attacks, too. hard to say what we're getting in exchange for the perf loss (hopefully security).

cc @paulirish (just in case).

[krausest]

@leeoniya The os patch level and drivers surely advanced between Chrome 77, 78 and 79 results, yet all runs were with mitigations=off.

But I took a closer look when Chrome 77 results were bad. I downgraded then chrome to 75 and 76 and confirmed the regression there (see #641).

I haven't found an easy way to downgrade chrome on ubuntu so I havent repeated it for chrome 78 and chrome 79 yet - if anyone knows better please let me know and I'll perform the downgrade.

[ryansolid]

Yeah 75 was a good one.. but if you go back further it just goes in cycles. I'd need to look it up for exact versions but there were some pretty slow versions in the early 70s.
What I wonder more about is just how close things are. Solid Signals while not showing statistical variability had a very poor run when the stock version is just pure overhead and scored generally better. At this point on any given day one of the vanilla variations, solid-signals, mikado, sinuous can have a run that puts them +-0.03 (which is nothing). I've been trying to determine if there is any correlation between certain chrome slowdowns. It does seem certain libraries do better at different times. Like Surplus,DomC, and Stage0 haven't been updated in a year and as browser slows down their stock rises. However in the past this usually saw a decrease in other areas not seeing now. It more feels like a bit of a round robin where chrome/v8 is updating perf in certain areas at the cost of others, until they get around to those again.
EDIT I was talking about yesterdays results. Today are different. I suppose sort of to the same point.

[krausest]

I couldn't downgrade chrome on ubuntu nicely, so i decided to try to downgrade on windows (from https://www.slimjet.com/chrome/google-chrome-old-version.php), which works better. I only had time to run two versions so I took Chrome 75 and the current version and checked that the performance difference is not due to system configuration.

Chrome 75 (slimjet installer):
result vanillajs-keyed_01_run1k.json min 101.646 max 127.522 mean 108.92739999999999 median 105.345 stddev 9.307251154043524

Chrome 79 (google installer):
result vanillajs-keyed_01_run1k.json min 129.306 max 166.251 mean 145.4538 median 145.45850000000002 stddev 10.291295089648443

As a side note: Chrome 79 reports two paint events whereas previous reported only a single paint event (but they are close enough not to matter in terms of difference to chrome 75).

[krausest]

Here's the timeline for two clicks on append 1,000 rows:
Chromium 75 on Ubuntu:

Chrome 79 on Ubuntu:

[krausest]

When testing manually with the timeline I often get the results above for Chrome 79. But sometimes there's a much quicker run like that:

I haven't seen such fast runs with my test driver though (maybe caused by code caching?)

[krausest]

At the same time Chromium 75 can occasionally report values like that:

Scripting time seems to vary a lot when running manually. I hope I find the time to take a look at it tomorrow.

[krausest]

Just for the record. On ubuntu the test driver extracts the following from the timeline:
Chromium 75:
result vanillajs-keyed_01_run1k.json min 84.043 max 100.55 mean 90.1799 median 88.407 stddev 6.019225909072064
Chrome 79:
result vanillajs-keyed_01_run1k.json min 108.613 max 131.918 mean 121.7578 median 122.3995 stddev 7.8771919390379495

[krausest]

Here are some more numbers extracted manually from the timeline (open chrome + dev tools, start profiling and reload page then click on append 1,000 rows and extract the duration from the timeline like in the screenshots above):

Chromium 75	Chrome 79
119	133
111	133
83	133
112	131
119	135
81	137
110	137
113	129
106	136
116	106
mean 107	mean 131
std dev 13,7759855465145	std dev 9,15302014516399

Observations:

• Chromium 75 is faster than Chrome 79 using this manual testing method too
• Some of the fastest results for chromium 75 match nicely the fastest results from the benchmark driver.
• The slower results for chromium 75 were not observed with the benchmark driver. In consequence the standard deviation is much higher for the manual test.
• Chrome 79 results were on average slower for the manual test, min and max look similar to the benchmark driver.
• The difference between slow and fast results seems to be mostly the scripting duration which varies between ~6 and ~30 msecs. Maybe this is more consistent for the benchmark driver?

[leeoniya]

haven't done extensive perf testing in Chrome 80 yet, but initial results seem to indicate a continued downward perf trend. my baseline bench [1] went from 42ms -> 55ms init (with significant variation).

meanwhile, Firefox is at 33ms with little-to-no variation.

might be interesting to see what can be gathered via https://github.com/microsoft/playwright, though this would be quite the undertaking.

[1] https://github.com/leeoniya/uPlot#performance

[krausest]

Results for Chrome 80 are now published
At first I thought it wasn't much worse than chrome 79, but this charts tells something different.

Do you have some good pointers about getting performance data in firefox?

[krausest]

I opened a chrome bug some time ago: https://bugs.chromium.org/p/chromium/issues/detail?id=1035938
Hope they find the time to take a look at it (and I hope even more it's not my mistake).

[leeoniya]

Do you have some good pointers about getting performance data in firefox?

no experience getting automated profiles, but they have some neat stuff in the works: https://profiler.firefox.com/, https://imgur.com/a/3fXISLP. unfortunately, it was not originally slated to be integrated into devtools. but i had a discussion [1] with them and think it lit a fire under some butts [2]. we'll see what comes of it.

[1] https://old.reddit.com/r/javascript/comments/ecyhmr/mozilla_survey_how_can_devtools_support_you/fbez5c0/
[2] https://news.ycombinator.com/item?id=21980439

[digitarald]

(Joining from the Firefox side, sensing a 🦊 signal)

@leeoniya I always got 🔥, maybe not where you tried to light it 😏 – glad you hear that you are excited about it coming! We are on track to get it into DevTools in H1 and hopefully all the way to release.

The easiest way to record a performance profile that can be loaded into profiler.firefox.com would be startup profiling; which means you'll get data as well from Firefox startup; but with performance.mark you could add data to slice the profile by later.

https://profiler.firefox.com/docs/#/./guide-startup-shutdown explains to enable env variables. MOZ_PROFILER_STARTUP=1 should get you started.

[leeoniya]

@RUSshy yeah i had same concerns above: #683 (comment). is there any way to get reproducible builds of Chrome rather than Chromium? if not, then the whole exercise might not be too meaningful. FWIW, Chromium builds [1] have always been slower for me than Chrome Stable releases.

[1] https://chromium.woolyss.com/

[krausest]

Repeated today:

Chromium	75.0.3770.90
run	sum	scripting	rendering
1	104	28	74
2	110	30	79
3	87	10	72
4	103	28	74
5	80	6	72
6	111	28	82
7	106	28	77
8	113	30	82
9	111	26	83
10	88	6	72
avg	101.3	21.91	76.67

(Yes - the 6 msecs were really reported in two runs!)

Chromium	80.0.3987.87
run	sum	scripting	rendering
1	138	30	102
2	141	31	104
3	123	17	104
4	128	31	96
5	135	30	102
6	130	28	99
7	130	30	97
8	135	32	100
9	137	31	105
10	131	33	95
avg	132.8	29.3	100.4

[ryansolid]

Sigh... I miss Chrome 75. Here's to Chrome 81 hopefully reversing this trend (not that I've had the chance to test it yet).

[leeoniya]

well i finally got some movement by floating it to the HN front page:

https://news.ycombinator.com/item?id=22267547

looks like they root-caused it [1] to rolling out LayoutNG:

https://developers.google.com/web/updates/2019/06/layoutNG
https://www.chromium.org/blink/layoutng

but there's been no updates since late feb. ¯\(ツ)/¯

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=1035938#c20

[krausest]

Results for Chrome 81 are published: https://krausest.github.io/js-framework-benchmark/2020/table_chrome_81.html

Please note that chrome 81 ran on Ubuntu 20.04-beta, such that the chart now includes the OS used:

One can see the positive effect Ubuntu 20.04 (maybe rather Gnome 3.36) had and that chrome 81 seems slower once again (at least for this particular benchmark)...

[krausest]

Results for chrome 83 are up:
https://krausest.github.io/js-framework-benchmark/2020/table_chrome_83.html
Looks much better than the last few versions. I even had to adjust the y axis to show the enormous improvement for remove rows.

[ryansolid]

Hmm more or less back to Chrome 77 other than remove row.

Remove row is interesting. Keyed moving the rows up is more performant than non-keyed replacing the data and removing the last row. Something must have fundamentally changed in how Chrome handles layout changes.

Memory is also down to unprecedented levels. I feel like it was only a year ago most libraries were at about 3mb ready memory. Now 80% of libraries are below 1.4mb. Even some of the worst libraries on memory are at about 50% of what they were (Like KnockoutJS was over 20mb on some tests now tops at 11.8mb)

[leeoniya]

Memory is also down to unprecedented levels.

i think they've been focusing a lot on reducing v8's/Chrome's mem footprint, since Chrome is famous for its RAM hunger.

https://v8.dev/blog/pointer-compression

[ryansolid]

@leeoniya That makes sense.

Although I wonder if there is a shelf. Look what has happened to Benchmark 7 the 10k rows. Most of the performant libraries more or less stayed the same. But the others dove right off the deep end. Like things that used to be 1.8 seconds are now 4 seconds. Those results seem to have large +- as well. I wonder if it is the browser or just the test driver. I guess the issue with React Djinn could be related.

[krausest]

I decided to withdraw the results for chrome 83 due to #741 .
So far I've no idea what causes the issue.

[brainkim]

One data point from a person who is actively working with the js-framework-benchmark right now: I’ve noticed is that the “shelf” is definitely related to the framework implementation. For instance, I’ve been playing around with an optimization where I try to group DOM mutations as tightly as possible; in other words, I first set up the tree of virtual nodes, and then actually turn this virtual DOM tree into actual DOM nodes in a separate pass, versus both setting up the tree and turning them into DOM nodes in the same pass. I initially did this just so I could identify bottlenecks in the flame graph but it also turned out to be faster to group DOM updates as tightly as possible, probably because it reduces the number of minor GCs somehow. When I updated to Chrome 83 from 81, it turned out this is the optimization which keeps me from falling off the performance cliff (fourth column is my framework 😓).

Virtual DOM updates and DOM updates separate.

Virtual DOM updates and DOM updates together.

I might have to back out of the separate DOM updates optimization because it’s causing weird race conditions with async code 😭 Just wanted to report in that variance aside, there could be actual framework related reasons for the change in the rowlots test.

[krausest]

Chrome 87 is reportedly very fast. In this benchmark almost as fast as chrome 75... (to be fair, remove it really fast)

Results for chrome 87 are published.

[krausest]

Chrome 88 results are up https://krausest.github.io/js-framework-benchmark/2021/table_chrome_88.0.4324.96.html
Overall maybe a bit slower than Chrome 87 (or maybe some security updates made that whole laptop slower)

[krausest]

Chrome 89 results showing a bad trend...

[krausest]

Chrome 90 is very special. It looks like it's the first version that beats chrome 75 in every aspect and is really pretty fast. It looks so good that I've published the results only as unofficial 😄 and take counsel with my pillow:

I haven't read anything about much improved performance in Chrome 90. Did I miss something?

P.S: As you can see I switched to Fedora 34. I thought about running the benchmark on Fedora 33, but it had an hard X11 crash. I compared results from the unfinished Fedora 33 run and the completed Fedora 34 run and they are close, maybe with a small advantage for Fedora 34. But the effect above is not from the system upgrade, but from the chrome update)

[maxmilton]

Chrome 90 includes V8 release v9.0: https://v8.dev/blog/v8-release-90.

I haven't looked over the actual V8 git log but I believe it includes this particularly exciting improvement: https://v8.dev/blog/adaptor-frame.

[ryansolid]

It's awesome we are finally back at better than Chrome 75 performance across the board. It's been almost 2 years.

[krausest]

Chrome 91 results are in: https://krausest.github.io/js-framework-benchmark/2021/table_chrome_91.0.4472.77.html
Looks like there's a very small improvement for Chrome 91. Regarding https://blog.chromium.org/2021/05/chrome-is-faster-in-m91.html this seems to be a bit disappointing (let‘s hope the benefits show up in other places):

Running the benchmark wasn't completely flawless. I git several "The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed." errors and had to rerun the benchmark for those cases.

[krausest]

Chrome 92 results are in. Partial update is significantly slower than on chrome 92.
Lighthouse was updated to 8.1 and it reports quite different results for script bootup time that I don't make much sense of (vanillajs gets a pretty bad result).
Here's the usual chart:

I'm looking for feedback for those results - I plan to publish them as official results tomorrow.

Results are published: https://krausest.github.io/js-framework-benchmark/2021/table_chrome_92.0.4515.107.html
I temporarily disabled script bootup time.

[ryansolid]

Yeah that is just bizarre. One thing I didn't notice for years was the vanillajs version isn't even minified. It's literally just a hand written JS file which is why other implementations come close in size. When you minify it Vanilla is clearly smaller. But it also suggests to me there is not much reason for it to be slower than anything else. It does create some DOM nodes on initial execution to make the templates that will be cloned but so do a number of other frameworks not showing the slow startup.

[krausest]

I'll have to check whether the chrome or the lighthouse update caused that behaviour. But I'm afraid I won't be able to take time to analyze this the next two months.

[lukechu10]

For some reason, Sycamore's results in Chrome 92 are way better than in Chrome 91, especially for create 10000 rows. It has improved by about 3x. It used to be the Major GC that took up a lot of time. Maybe the GC is lazier? Not quite sure what happened.

[leeoniya]

Maybe the GC is lazier? Not quite sure what happened.

yes, it's looking lazier since 91. see also #916

[krausest]

Results for chrome 93 are available: https://krausest.github.io/js-framework-benchmark/2021/table_chrome_93.0.4577.63.html
The memory table is currently hidden in the official results due to #916.
Results look pretty much like chrome 92. Sadly it didn't magically solve the memory issue since chrome 91.

[leeoniya]

we should probably close out https://bugs.chromium.org/p/chromium/issues/detail?id=1035938 at this point?

[bfgeek]

Hi,

I was going through our backlog of Blink>Layout bugs a couple of months ago, saw this ( https://bugs.chromium.org/p/chromium/issues/detail?id=1035938), and wanted to give a little context around all the step changes in performance over the past 1-2 years or so.

Apologies I missed the bug the first time round (it was inadvertently taken out of the triage queue).

The initial performance regression was likely (in part) due to us removing a very particular type of cache. Historically when measuring inline content (text) browsers will measure on a word-by-word basis, to build up a paragraph of text. Essentially we'd build up a: Map<TextRun, ShapeResult> where the ShapeResult has the width of the word.

This design is simple, but has its drawbacks - it cannot handle some fonts (e.g. if the kerning table has spaces in it), or handle complex text particularly well. This means that users who use these languages/scripts (Thai, Persian) typically have had significantly worse performance as opposed to Latin scripts (even with Latin scripts, some languages e.g. German make the word cache less effective).

Today Blink will shape the whole paragraph of text once which we found has higher throughput on average, and for most pages faster than checking the word-shape-cache for each word (more on this in a second). We don't really attempt to cache any shape results from harfbuzz (except when appending text).

We found that some microbenchmarks did get slower however. We found that these microbenchmarks were typically using a fixed small dictionary for words. This means that the word shape cache was typically super effective at these microbenchmarks, however the new system less so as we'd shape everything on a per paragraph basis (everything within a table cell in this case).

For this specific benchmark I believe the word dictionary is relatively small ~30-50 words for the 2nd column. The numbers (1st column) is better but again if removing/inserting the number would hit the same spot in the cache.

Removing this cache is a tradeoff (some pages/webapps do use a lot of the same repeated words!) but the speedup and complexity reduction for other scripts was significant (as in sometimes a couple of orders of magnitude in performance improvement).

Some additional slowdown was likely due to the "Legacy<->NG" bridge we build in our layout system, transitioning between the two layout systems was more expensive that if it kept within the same system.
For tables specifically initially every table cell would have a bridge inserted.

The performance improvement around the ~M86 range was likely due to some optimizations we made to inline layout (collapsing things into a flat list rather than a tree heirarching for inline content).

Finally the performance improvement around M91 was likely due to us moving tables completely onto the new layout system. This removed the bridge (not that much of an improvement by itself). Tables are historically very complicated, we have some additional caching from our new system which likely helps slightly as well as just higher raw throughput in the table layout code due to better data-structures.

(there are some really complex optimizations that we could add specifically for tables, which would likely help significantly the remove/create/replace benchmarks, but these would be super difficult to maintain).

Unsure what was the cause for the significant drop in "remove" around M83.

Next year there are some more improvements that we'll be able to take, but nothing super significant (e.g. ~5% improvements here and there).

In the next couple of releases there may be a regression due to more rendering data-structures (style, layout, paint) moving onto our garbage collector for security reasons. We've seen this regress some microbenchmarks on the order of ~10-15%. Typically this is due to some GC pauses, some of this will be eventually mitigated due to making the GC faster (marking off-thread etc).

Ian

[krausest]

Results for chrome 94 are updated. Please note that I still couldn't fix the memory issue #916 and thus I'm really thinking about rewriting the test driver with puppeteer.
As @bfgeek mentioned in his great reply above chrome 94 appears to be a bit slower than chrome 93:

[krausest]

Chrome 95 was released yesterday. Results are online

A bit slower for creating rows and definitely faster for replacing rows.
@Freak613 looks like select rows is a bit slower for 1more (I ran it a few times to make sure it wasn't a one time outlier)

[krausest]

Chrome 96: There's a continuing trend that chrome appears to becoma a little slower for this particular benchmark.
Results are online: https://krausest.github.io/js-framework-benchmark/2021/table_chrome_96.0.4664.45.html

[krausest]

Happy new year! This thread got pretty long so we'll continue 2022 results in #984