Replace test-collector-swift with JUnit upload by mokagio · Pull Request #25251 · wordpress-mobile/WordPress-iOS

mokagio · 2026-02-13T06:26:49Z

Summary

Replaces the test-collector-swift SPM package (runtime XCTest hook) with the test-collector Buildkite plugin that uploads JUnit XML files to Test Engine.

The SPM package hooks into XCTest at runtime, so Swift Testing tests are invisible to Test Engine.
The Buildkite plugin uploads JUnit XML files post-step, which works with any test framework.

The SPM dependency and all its supporting code (env injection, token exports, trainer calls) are removed.
The Buildkite plugin is added to the 3 test steps that have a Test Engine token.

Fixes AINFRA-1977

Testing

Let CI run on this PR
Verify all test steps pass
Check the Buildkite Test Analytics dashboard to confirm results are received

Posted by Claude (Opus 4.6) on behalf of @mokagio with approval.

wpmobilebot · 2026-02-13T06:40:29Z

📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.

	App Name	WordPress
	Configuration	Release-Alpha
	Build Number	`30990`
	Version	`PR #25251`
	Bundle ID	`org.wordpress.alpha`
	Commit	`c09fdcb`
	Installation URL	5sbjkvpprh5q8

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

wpmobilebot · 2026-02-13T06:40:31Z

📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.

	App Name	Jetpack
	Configuration	Release-Alpha
	Build Number	`30990`
	Version	`PR #25251`
	Bundle ID	`com.jetpack.alpha`
	Commit	`c09fdcb`
	Installation URL	02913jhi7oh58

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

wpmobilebot · 2026-02-13T06:55:17Z

🤖 Build Failure Analysis

This build has failures. Claude has analyzed them - check the build annotations for details.

Copilot

Pull request overview

This PR replaces the test-collector-swift SPM package (runtime XCTest hook) with the Buildkite test-collector plugin that uploads JUnit XML files to Test Engine. This migration enables test analytics for Swift Testing tests, which were invisible to the runtime hook approach.

Changes:

Removed test-collector-swift SPM dependency and all supporting code
Added Buildkite test-collector plugin (v1.11.0) configuration to pipeline
Updated test reporting to use JUnit XML format with standard filename

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`fastlane/lanes/build.rb`	Removed TEST_ANALYTICS_ENVIRONMENT constant, inject_buildkite_analytics_environment method, trainer call, and buildkite_ci? helper; changed output_types to 'junit'
`Tests/WordPressKitTests/WordPressKitTests/Tests/TestCollector+Constants.swift`	Deleted file containing TestCollector API host constant
`Tests/WordPressKitTests/WordPressKitTests/Tests/RemoteTestCase.swift`	Removed BuildkiteTestCollector import and updated stubAllNetworkRequestsWithNotConnectedError to stub all requests
`Modules/Package.swift`	Removed test-collector-swift package dependency from three test targets
`Modules/Package.resolved`	Removed resolved test-collector-swift package entry
`.buildkite/shared-pipeline-vars`	Added TEST_COLLECTOR_PLUGIN_VERSION and TEST_COLLECTOR environment variables
`.buildkite/pipeline.yml`	Added test-collector plugin configuration to WordPress Unit Tests, Jetpack UI Tests (iPhone), and Jetpack UI Tests (iPad)
`.buildkite/commands/run-unit-tests.sh`	Removed BUILDKITE_ANALYTICS_TOKEN export and updated annotate_test_failures file path to report.junit
`.buildkite/commands/run-unit-tests-reader.sh`	Removed commented-out test analytics token configuration
`.buildkite/commands/run-ui-tests.sh`	Removed BUILDKITE_ANALYTICS_TOKEN export logic and updated annotate_test_failures file path to report.junit

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

AliSoftware · 2026-02-18T16:19:00Z

Not sure why the Prototype builds error with:

Error: Could not find action, lane or variable 'sentry_upload_dsym'

While I'd think that it worked before?

Also, there are some Rubocop lint errors, (I'm not sure why they didn't get inlined as PR comment though… ):

fastlane/lanes/build.rb#L130: Layout/EmptyLinesAroundBlockBody: Extra empty line detected at block body end.

AliSoftware

Approving to unblock, but left some minor suggestion about artifact naming + a recommendation for keeping fail_build: true, so disabled auto-merge to let you the opportunity to fix them—in addition to the 3 test failures on CI—before merging.

AliSoftware · 2026-02-18T16:25:39Z

.buildkite/pipeline.yml

+        plugins:
+          - $CI_TOOLKIT_PLUGIN
+          - ${TEST_COLLECTOR}:
+              files: build/results/report.junit


It would be nice if the JUnit report files could have a different/unique name for WordPress vs Jetpack and UI vs Unit tests (like they had before), so that once they're uploaded as build artifacts (builds/results/*) they are easy to distinguish?

Since we have one step/job for each of those technically this shouldn't be an issue—each step will have its own separate artifacts attached to it, so it's not like one would overwrite the other across jobs—but in case someone needs to call buildkite-agent artifact download at some point to download them, it'd be nicer if they wouldn't need to specify the --step <step_key_name> to be able to differentiate / indicate which of they they want to download… especially since some of those steps don't have a key to allow to identify them anyway. While if those artifacts had unambiguous names, different across jobs, the --step wouldn't be needed to begin with.

AliSoftware · 2026-02-18T16:27:46Z

fastlane/lanes/build.rb

      max_concurrent_simulators: CONCURRENT_SIMULATORS
    )

-    trainer(path: lane_context[SharedValues::SCAN_GENERATED_XCRESULT_PATH], fail_build: true)


If we remove this line, shouldn't we then restore the fail_build: argument of run_tests to true on line L138R125 to keep the behavior that failed tests should mark the CI build as failed?

Indeed, before run_tests was using fail_build: false because we then called trainer as a subsequent action, with its fail_build: true. But now that we don't call trainer (I think because it's implicitly called by run_tests when output_types contains junit, iinm), to keep the behavior we want to keep fail_build: true somewhere.

[EDIT] Added an adhoc GitHub suggestion for this change above

Addressed in cc8ae22 — removed fail_build: false so it defaults back to true.

Posted by Claude Code (Opus 4.6) on behalf of @mokagio with approval.

fastlane/lanes/build.rb

AliSoftware · 2026-02-18T16:34:00Z

fastlane/lanes/build.rb

      result_bundle: true,
-      output_types: '',
+      output_types: 'junit',
      fail_build: false,


Suggested change

fail_build: false,

fail_build: true,

mokagio · 2026-02-19T06:23:59Z

@AliSoftware

Not sure why the Prototype builds error with:

Error: Could not find action, lane or variable 'sentry_upload_dsym'

While I'd think that it worked before?

Looks like #25232 merged a major version bump of the plugin that broke the prototype build.

sentry_upload_dsym action has been removed: This action has been deprecated in favor of sentry_debug_files_upload. Users should migrate to sentry_debug_files_upload with the appropriate path parameter. (getsentry/sentry-fastlane-plugin#375) — https://github.com/getsentry/sentry-fastlane-plugin/releases/2.0.0

And #25264 fixed it.

--- Generated with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The default is `true`, which is what we want now that `scan` handles failures directly via JUnit instead of delegating to `trainer`. --- Generated with the help of Claude Code, https://claude.ai/claude-code Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-authored-by: Olivier Halligon <olivier.halligon@automattic.com>

sonarqubecloud · 2026-02-19T06:38:18Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

mokagio · 2026-02-19T07:57:54Z

@AliSoftware thank you for the JUnit report file name suggestion. I'm going to merge without it and think about it. Just because I'd like to find a neat way not to duplicate the file name between pipeline and Fastfile

AliSoftware · 2026-02-19T08:12:34Z

@AliSoftware thank you for the JUnit report file name suggestion. I'm going to merge without it and think about it. Just because I'd like to find a neat way not to duplicate the file name between pipeline and Fastfile

Maybe another approach is to be ok with them having the same name… but ensuring all pipeline steps that use those artifacts have a key so that we can reliably use buildkite-agent artifact download report.junit --step {key}?

mokagio added the [Type] Task label Feb 13, 2026

mokagio added this to the 26.8 milestone Feb 13, 2026

AliSoftware requested a review from Copilot February 13, 2026 18:42

Copilot started reviewing on behalf of AliSoftware February 13, 2026 18:43 View session

Copilot AI reviewed Feb 13, 2026

View reviewed changes

mokagio enabled auto-merge February 16, 2026 02:08

AliSoftware disabled auto-merge February 18, 2026 16:31

AliSoftware approved these changes Feb 18, 2026

View reviewed changes

AliSoftware reviewed Feb 18, 2026

View reviewed changes

mokagio mentioned this pull request Feb 19, 2026

Replace test-collector-swift with JUnit upload woocommerce/woocommerce-ios#16668

Merged

mokagio and others added 3 commits February 19, 2026 17:24

Replace test-collector-swift with JUnit upload

1663171

--- Generated with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remove extra empty new line

c09fdcb

Co-authored-by: Olivier Halligon <olivier.halligon@automattic.com>

mokagio force-pushed the ainfra-1977-migrate-all-apple-codebase-from-the-test-engine-swift branch from c4a24e9 to c09fdcb Compare February 19, 2026 06:35

mokagio added this pull request to the merge queue Feb 19, 2026

Merged via the queue into trunk with commit d5eae8d Feb 19, 2026
27 of 32 checks passed

mokagio deleted the ainfra-1977-migrate-all-apple-codebase-from-the-test-engine-swift branch February 19, 2026 08:13

Conversation

mokagio commented Feb 13, 2026

Summary

Testing

Uh oh!

wpmobilebot commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wpmobilebot commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wpmobilebot commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 Build Failure Analysis

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

AliSoftware commented Feb 18, 2026

Uh oh!

AliSoftware left a comment

Choose a reason for hiding this comment

Uh oh!

AliSoftware Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

AliSoftware Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mokagio Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AliSoftware Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

mokagio commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Feb 19, 2026

Quality Gate passed

Uh oh!

mokagio commented Feb 19, 2026

Uh oh!

AliSoftware commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wpmobilebot commented Feb 13, 2026 •

edited

Loading

wpmobilebot commented Feb 13, 2026 •

edited

Loading

wpmobilebot commented Feb 13, 2026 •

edited

Loading

AliSoftware Feb 18, 2026 •

edited

Loading

mokagio commented Feb 19, 2026 •

edited

Loading

AliSoftware commented Feb 19, 2026 •

edited

Loading