Support XCTest Plan use cases in tuist test

[shahzadmajeed]

What happened?

Currently, we are seeing following issues with tuist test when we switch to XCTest Plans from normal test bundles.

Issue 1 - Tests scheme is not identified by tuist test command

tuist test is unable to identify main/default scheme which was working fine when test bundles were used directly (before migrated to XCTest Plans). Tuist seems to include another (new) scheme in this case which can work fine but consumers have to update CI to adopt new scheme. It would be great if it can continue to work with existing scheme.

Notes:

1. xcodebuild can identify the original scheme fine
2. Original scheme can also be spotted in generated project and it works from within the IDE as well

Solution:

Issue # 1 is just a bug and can be solved in current workflow

Issue 2 - Improve tuist test to support XCTest Plans arguments

XCTest Plans can be setup with multiple configurations but xcodebuild will run all tests in a test plan against each configuration which could be time wasting in many cases. We would want to only run a default or selected configuration for a test plan. Currently xcodebuild only supports one -testPlan argument which means you cannot run multiple plans at a time so it simplifies our use cases

So, we need to support:

1. -testPlan test plan that you want to run
2. -only-test-configuration a list of configurations in specified test plan that you want to run (note that we will have to repeat -only-test-configuration for each config when passing this param to xcodebuild internally)
3. -skip-test-configuration a list of configurations in specified test plan that you want to skip (note that we will have to repeat -skip-test-configuration for each config when passing this param to xcodebuild internally)

Note: Normally users would either want to use -only-test-configuration or -skip-test-configuration but we need to support both.

Solution:

Issue # can be solved in 2 ways (suggestions are welcome)

1. Allow a flexible additionalParams param in tuist test which can be passed to xcodebuild as is. This can give users a greater control over the build process but can also introduce conflicts with Tuist's internal logic. For example, caching mayn't work as expected
2. A stricter approach would be to allow all the needed params in tuist test command and configure xcodebuild internally. This is less controlling for users and might require future maintenance or adoption of new use cases

It was suggested in this slack thread that we can use approach # 1.

Note: Xcode 14.3 will enable test plans by default and will convert schemes to provide a default plan which means we should probably be ready for this migration sooner than later.

How do we reproduce it?

1. Convert test bundles into a XCTest Plan
2. Try to run tuist test command for existing scheme - you will see that scheme is not found even though it exists in generated project
3. tuist test currently don't support use cases in issue # 2

Error log

 >>/TuistWorkspace  testing/xctest_plans  tuist test MyApp-AllTests -p Projects/MyApp         
Generating project for testing
Generating workspace MyApp.xcworkspace
Generating project PlatformSDK
Generating project GamingBase
Generating project combine-schedulers
Generating project swift-case-paths
Generating project Kingfisher
Generating project swift-clocks
Generating project swift-composable-architecture
Generating project swift-collections
Generating project Sportsbook
Generating project Difference
Generating project swift-identified-collections
Generating project swift-custom-dump
Generating project swift-dependencies
Generating project SnapshotTesting
Generating project swiftui-navigation
Generating project xctest-dynamic-overlay
Couldn't find scheme MyApp-AllTests. The available schemes are: MyApp-Workspace, MyApp-Workspace.
Consider creating an issue using the following link: https://github.com/tuist/tuist/issues/new/choose

macOS version

12.0

Tuist version

3.16.0

Xcode version

14.2

[github-actions]

Hola 👋,

We want to inform you that the issue has been marked as stale. This means that there hasn't been any activity or updates on it for quite some time, and it's possible that it may no longer be relevant or actionable.
If you still believe that this issue is valid and requires attention, please provide an update or any additional information that can help us address it. Otherwise, we may consider closing it in the near future.
Thank you for your understanding.

[github-actions]

Hola 👋,

We want to inform you that we have decided to close this stale issue as there hasn't been any activity or response regarding it after marking it as stale.

We understand that circumstances may have changed or priorities may have shifted, and that's completely understandable. If you still believe that this issue needs to be addressed, please feel free to reopen it and provide any necessary updates or additional information.

We appreciate your understanding and look forward to your continued contributions to the project.

Thank you.

[stephanecopin]

This is not stale and still being worked on in #5135, reopening.