Review Performance Regressions in WP 6.6 beta

[joemcgill]

After WP 6.6 beta 1 was packaged, @adamsilverstein performed the first set of benchmarks for this release and reported results here.

I ran benchmarks comparing WordPress 6.6 Beta 1 and the latest released 6.5 branch (6.5.3). I took measurements with the twentytwentyone (TT1) theme using en_US locale, both with memcached disabled and enabled, then again with twentytwentyfour (TT4).

This version of WordPress is showing a small - though not insignificant - regression in server rendering time from the previous release. Classic themes (TT1) show a ~5% regression (2ms slower) in TTFB and wp-total measurements, while Block themes (TT4) show a ~14% regression (10ms slower). LCP remains relatively stable for classic themes, while block themes show a 4-6% regression (with caching enabled/disabled).

These regressions suggest spending time investigating the underlying cause and potential fixes. As a side note, the metrics dashboard - https://www.codevitals.run/project/wordpress - doesn’t show the same regression even for similar metrics. A follow should be to investigate why and improve the consistency there.

Summarized data - [shared] Performance testing 6.6 beta 1 vs 6.5.3 (latest) June 4, 2024

Benchmark runs
TT1 no caching: https://github.com/adamsilverstein/compare-wp-performance/actions/runs/9375220810
TT1 with caching: https://github.com/adamsilverstein/compare-wp-performance/actions/runs/9375222140
TT4 no caching: https://github.com/adamsilverstein/compare-wp-performance/actions/runs/9375223280
TT4 with caching: https://github.com/adamsilverstein/compare-wp-performance/actions/runs/9375378961

We should investigate the cause of these regressions to see what issues can be identified and remediated prior to the final release. Additionally, investigation into why these regressions are showing up in release benchmarks but not in the metrics recorded by the automated performance workflow for each commit to WP Core.

Following initial investigation, follow-up tickets should be opened in Trac and/or the Gutenberg repo (whichever is appropriate) and can be added to the list of follow-up tasks below.

Tasks

• Spike: Identify causes of performance regressions in WP 6.6 beta 1
• Spike: Investigate compare-wp-performance script for errors
• HOLD Spike: Review discrepancy with the automated performance workflow

[joemcgill]

@adamsilverstein I've spent some time investigating the source of the performance regressions, and am thinking the automated benchmarks may be reporting inaccurate data and that WP 6.6 Beta actually has better performance numbers than 6.5.3.

WordPress/wordpress-develop benchmarks

Using the wordpress-develop environment, I've taken measurements at individual commits and saved the server timing metrics to this spreadsheet, which I've summarized below.

Metric	6.5.3	6.6 Beta 1
Response Time (median)	70.82	57.57
wp-before-template (median)	34.53	20.05
wp-template (median)	32.72	35.47
wp-total (median)	68.67	55.51

More details about the process

Process

For each benchmark, I would make sure that I had refreshed npm packages and run a new build at each commit (i.e., npm ci && npm run build:dev). I then ran benchmarks via our CLI using 100 iterations, (i.e., npm run research -- benchmark-server-timing --url http://localhost:8889 -n 100).

wp-config.php settings

define( 'WP_DEBUG', false );
define( 'SCRIPT_DEBUG', false );
define( 'SAVEQUERIES', false );
define( 'DISABLE_WP_CRON', true );

Commits checked

I ran benchmarks at the following commits where I was aware of the potential for performance impacts, along with the commit prior to a change.

• 9d0f7b9 - WP 6.5.3
• 52f010c - WP 6.5.4
• 7d606a3 - Disable autoloading for large options
• 22ad614 - Cache block the patterns
• d1d0f68 - Update autoload defaults in Core
• 6f6bd30 - Remove redundant WP_Theme_JSON::__construct calls
• 2546a74 - Package sync 1
• 2c81850 - Package sync 2
• c1dad7d - WP 6.6 Beta 1

@wordpress/env benchmarks

To double check, I also ran the benchmarks locally using a clean wp-env environment, and am seeing similar improvements to both server timing and CWV metrics:

Metric	6.5.3	6.6 Beta 1
Response Time (median)	42.39	39.33
wp-before-template (median)	21.65	14.34
wp-template (median)	18.92	23.13
wp-total (median)	40.71	37.39
FCP (median)	146.5	128.2
LCP (median)	147.6	128.2
TTFB (median)	102	80.65
LCP-TTFB (median)	48.2	46.95

Next steps

Assuming I've not made several mistakes here, I think our next step is to investigate why the automated benchmark script is reporting metrics that show a large regressions so we can fix that script to be more reliable going forward.

[joemcgill]

@adamsilverstein posted another set of benchmarks using the automated workflow in Slack:

I ran benchmarks comparing WordPress 6.6 Beta 1 and the latest released 6.5 branch (6.5.4). I took measurements with the twentytwentyone (TT1) theme using en_US locale, both with object caching disabled and enabled, then again with twentytwentyfour (TT4).

Similar to 6.6 beta 1, this version of WordPress shows a regression in server rendering time from the previous release.
With object caching disabled, Classic themes (TT1) show a ~5% regression (2ms slower) in TTFB and wp-total measurements; LCP remains relatively stable. Block themes (TT4) show a ~12% regression in TTFB (10ms slower) and a 14% regression (9 ms slower) for wp-total measurements. LCP shows a 4% regression (6ms). Running the tests with object caching enabled shows slightly reduced regressions.

Based on these tests, I would like the performance team to try to identify the underlying cause and suggest possible improvements. I won’t have time to work on this soon, so it would be great if someone else could look into it. Investigations could start by scanning large commits for potential issues or using a tool like git bisect to pinpoint the introduction of the regressions. As a reminder, the metrics dashboard - https://www.codevitals.run/project/wordpress - doesn’t show the same regression even for similar metrics. We should investigate why and improve the consistency here.

[mukeshpanchal27]

@adamsilverstein @joemcgill I did manual benchmark for 6.5.X and 6.6 beta 2.

Metric	6.5.X	6.6 Beta 2	Diff %	Diff abs.
wp-before-template (median)	52.96	44.72	-15.56%	-8.24
wp-template (median)	25.94	31.64	21.97%	5.70
wp-total (median)	78.91	76.45	-3.12%	-2.46

• wp-before-template is faster
• wp-template is slower
• wp-total is faster.

Find full result here.

[joemcgill]

Thanks, @mukeshpanchal27 that's similar to what I was seeing above. @adamsilverstein's most recent set of benchmarks shows the same performance characteristics for Twenty Twenty-Four. Performance prior to template rendering is improved, but the template rendering is slower. In his case, the regression to template rendering offsets the pre-rendering gains, which is why he's showing a regression on wp-total.

Seems like something new being added to the block rendering process is likely leading to regression.

[joemcgill]

Earlier today, @dmsnell reported the results of his own profiling in Slack.

that shows every 10 commits from 6.5 until trunk in wordpress/wordpress and this is rendering the home page, all repeat requests

Two possible commits to investigate:

• r57920: Options, Meta APIs: Use more sensible default for autoloading options...
• r58264: Block Themes: Add section styling via extended block style variations

The first of those is showing up as a performance improvement in most other places, so he is planning to re-run his profiles to confirm there wasn't a misconfiguration somewhere. The second looks like a potential area for investigation and is being worked on in https://core.trac.wordpress.org/ticket/61451.

[joemcgill]

I did a round of benchmarks locally comparing the autoload options commit with the commit before. For both, I used a fresh site to ensure clean runs and took 100 sample requests:

Before:

• wp-before-template (p50) – 31.97ms
• wp-template (p50) – 32.7ms
• wp-total (p50) – 65.52ms

After:

• wp-before-template (p50) – 29.66ms
• wp-template (p50) – 31.24ms
• wp-total (p50) – 60.92ms

I've got these constants set in my wp-config:

define( 'WP_DEBUG', false );
define( 'SCRIPT_DEBUG', false );
define( 'SAVEQUERIES', false );
define( 'DISABLE_WP_CRON', true );

And for comparison, I ran them again but this time didn't reset the DB between aa04408 and 7d606a3

Before:

• wp-before-template (p50) – 30.02ms
• wp-template (p50) – 31.05ms
• wp-total (p50) – 61.05ms

After:

• wp-before-template (p50) – 29.31ms
• wp-template (p50) – 30.57ms
• wp-total (p50) – 60.15ms

[mukeshpanchal27]

@joemcgill @adamsilverstein I did manual and GA benchmark for 6.5.X and 6.6 RC 4.

WordPress/wordpress-develop manual benchmarks

TT4 theme

Metric	6.5.X	6.6 RC 4
Response Time (median)	80.06	75.6
wp-before-template (median)	52.6	42.23
wp-template (median)	25.57	31.3
wp-total (median)	78.21	73.73

TT1 theme

Metric	6.5.X	6.6 RC 4
Response Time (median)	21.19	20.88
wp-before-template (median)	10.15	9.82
wp-template (median)	9.48	9.71
wp-total (median)	19.72	19.53

Benchmark runs

TT1 no cache: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/9950889179
TT4 no cache: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/9951185365
TT1 cache: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/9951332697
TT4 cache: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/9951419572

Both the workflow shows the regression in server timing matric wp-template for block theme. The classic theme didn't show any regression.

[adamsilverstein]

@joemcgill @adamsilverstein I did manual and GA benchmark for 6.5.X and 6.6 RC 4.

@mukeshpanchal27 thanks for running those tests!

Interesting that your manual testing doesn't really show the regressions that the automated testing seems to show. Did you set constants like Joe mentions above and run a bunch of iterations without doing anything else on your device?