Build/Test Tools: Publish PHPUnit timing metrics to CodeVitals

[lancewillett]

Trac ticket: https://core.trac.wordpress.org/ticket/65887

What this changes

The canonical PHPUnit job publishes six aggregate timing metrics to the existing public CodeVitals dashboard after every push to trunk:

• PHP 8.5;
• MySQL 8.4;
• single site;
• default domain and cache configuration.

The publisher reads the existing JUnit report immediately after the main test suite, before later test-group runs replace the file. It streams the XML, creates a small JSON payload, and sends no test names or per-test records.

This complements #13070. That pull request preserves the raw JUnit report for detailed analysis; this one adds the visible trend layer.

Metrics

Key	Display name	Unit	Default
phpunit-suite-time	PHPUnit main suite time	s	Visible
phpunit-p95-test-time	PHPUnit test time (p95)	ms	Other
phpunit-p99-test-time	PHPUnit test time (p99)	ms	Other
phpunit-max-test-time	PHPUnit maximum test time	ms	Other
phpunit-tests-over-500ms	PHPUnit tests over 500 ms	tests	Other
phpunit-tests-over-1s	PHPUnit tests over 1 s	tests	Other

Keeping one metric visible by default avoids crowding the existing dashboard. The other five remain available from the Other selector for diagnosis.

The definitions must exist in CodeVitals before the first submission. Suggested priorities are 100–105 in the order above, with a regression noise floor of 0 until trunk runs establish a baseline.

The workflow also verifies that CodeVitals accepted all six metrics. A missing definition fails the step instead of silently dropping data.

How to validate

After the first trunk push following merge, open:

https://codevitals.run/public/WordPress/wordpress-develop/metrics

The PHPUnit main suite time card should appear. Select it to view the trend by commit. Use Other to add the percentile, maximum, and slow-test-count metrics.

The parser can be checked locally with:

npm run test:php -- --filter Tests_Includes_JUnit_Timing_Metrics

A previously captured CI report containing 30,871 timed test cases produced:

• suite: 281.19193 seconds;
• p95: 49.723 milliseconds;
• p99: 100.294 milliseconds;
• maximum: 3,446.888 milliseconds;
• over 500 milliseconds: 31 tests;
• over 1 second: 19 tests.

An authenticated CodeVitals submission remains a pre-merge validation item. The token stays in the existing CODEVITALS_PROJECT_TOKEN Actions secret and is never available to local or pull-request runs.

Testing

• Focused PHPUnit tests passed: 4 tests, 6 assertions.
• PHP syntax checks and WordPress PHP coding standards passed.
• actionlint passed for both changed workflow files.
• The command-line publisher returned all six numeric metrics from both focused and full CI JUnit reports.
• The CodeVitals endpoint returned the expected authentication error for an invalid test token.

[github-actions]

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

• All changes will be lost when closing a tab with a Playground instance.
• All changes will be lost when refreshing the page.
• A fresh instance is created each time the link below is clicked.
• Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
  it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

[lancewillett]

@johnbillion Thanks for suggesting CodeVitals on https://core.trac.wordpress.org/ticket/65887#comment:4.

CodeVitals requires GitHub repository admin access to manage metric definitions. Could you add these six definitions for WordPress/wordpress-develop?

Key	Display name	Unit	Priority	Visible by default	Noise floor
phpunit-suite-time	PHPUnit main suite time	s	100	Yes	0
phpunit-p95-test-time	PHPUnit test time (p95)	ms	101	No	0
phpunit-p99-test-time	PHPUnit test time (p99)	ms	102	No	0
phpunit-max-test-time	PHPUnit maximum test time	ms	103	No	0
phpunit-tests-over-500ms	PHPUnit tests over 500 ms	tests	104	No	0
phpunit-tests-over-1s	PHPUnit tests over 1 s	tests	105	No	0

The service silently drops unknown metric keys. This PR verifies that all six were accepted, so the scheduled publishing step will fail clearly if any definition is missing.

Once these exist, I can complete the authenticated submission and verify the public dashboard: https://codevitals.run/public/WordPress/wordpress-develop/metrics

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props lancewillett, desrosj.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[desrosj]

Some small questions and change requests.

[lancewillett]

Thanks @desrosj -- fixed pushed bfd9637

is there any reason why we can not submit test results for every commit?

I originally limited this to weekly runs for storage and consistency, but updated it to publish after every push to trunk based on your feedback.

PR runs remain excluded because their branch-specific results are less comparable—and would create a LOT more storage that isn't that helpful for stats. At least, not now.

[lancewillett]

@adimoldovan Curious your take on the approach. Is there a better way to store the test run results and display it in a dashboard somewhere?