Add build-only mode for Xcode 15.x to fix Swift Testing macro errors

[leogdion]

Xcode 15's xcodebuild cannot resolve external macro plugins from the swift-testing package. Use build-only mode for Xcode 15.x to validate compilation while running full tests on Xcode 16.4+ and Ubuntu.

This keeps Xcode 15.x in the CI matrix for build validation while avoiding the macro resolution errors during test execution.

---

Perform an AI-assisted review on

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/xcode15-build-only

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[claude]

PR Review: Add build-only mode for Xcode 15.x

Summary

This PR addresses a legitimate issue with Xcode 15's inability to resolve external macro plugins from the swift-testing package by enabling build-only mode for Xcode 15.x builds while maintaining full test execution on newer Xcode versions.

---

✅ Positives

1. Targeted Fix: The change is minimal and surgical - only adding the build-only parameter conditionally for Xcode 15.x versions.

2. Pattern Consistency: The implementation follows the existing pattern already used for skip-package-resolved, making it easy to understand:

   build-only: ${{ contains(matrix.xcode, 'Xcode_15') }}
   skip-package-resolved: ${{ contains(matrix.xcode, 'Xcode_15') }}

3. Maintains CI Coverage: The approach preserves Xcode 15.x in the CI matrix for build validation, which is valuable for ensuring compilation compatibility across Xcode versions.

4. Known Issue: Swift Testing's macro support requires Xcode 16+, so this is a reasonable workaround for a known toolchain limitation.

---

🔍 Considerations & Questions

1. Test Coverage Gap: While the build-only approach validates compilation, it means no tests are executed for the Xcode 15.x configurations. This creates a potential blind spot if there are runtime issues specific to those iOS/watchOS/tvOS/visionOS versions.

   Affected matrix entries (from the diff context):

   • Xcode 15.0.1 configurations
   • Xcode 15.4 configurations
   • Multiple platform types (default, macos, ios, watchos, visionos)

   Questions:

   • Are tests still running on Ubuntu/Linux for Swift 5.9/5.10? (These might use the same older toolchain)
   • Could runtime behavior differ between iOS 17.x (Xcode 15) and iOS 18.x (Xcode 16)?

2. Coverage Reporting Impact: Lines 273-280 show coverage processing and upload steps. With build-only: true, these steps might:

   • Fail (if no coverage files are generated)
   • Be skipped (if the swift-build action handles this)
   • Upload empty/partial coverage

   Recommendation: Consider adding a conditional if: ${{ !contains(matrix.xcode, 'Xcode_15') }} to the coverage steps to avoid potential failures.

3. Documentation: The PR description is clear, but consider:

   • Adding an inline comment in the workflow file explaining why Xcode 15 uses build-only mode
   • Documenting the plan for eventually removing Xcode 15 support (if applicable)

4. Matrix Redundancy: I notice there are duplicate Xcode 15.4 entries in the matrix (lines 62-64 and 82-85 in the full file). One has no type specified, the other has type: macos. Is this intentional?

---

🔒 Security & Performance

• Security: No concerns. This is a CI configuration change only.
• Performance: Actually improves CI performance by skipping test execution for Xcode 15.x builds (though this is a side effect, not the goal).

---

📝 Suggestions

1. Add conditional to coverage steps to prevent potential failures:

   - name: Process Coverage
     if: ${{ !contains(matrix.xcode, 'Xcode_15') }}
     uses: sersoft-gmbh/swift-coverage-action@v4
   
   - name: Upload Coverage
     if: ${{ !contains(matrix.xcode, 'Xcode_15') }}
     uses: codecov/codecov-action@v4

2. Add inline comment for future maintainers:

   # Xcode 15.x cannot resolve Swift Testing macros; use build-only mode
   build-only: ${{ contains(matrix.xcode, 'Xcode_15') }}

3. Consider deprecation timeline: If Xcode 15 support isn't critical, consider documenting when it might be removed from the CI matrix.

---

✅ Recommendation

APPROVE with minor suggestions. The change is sound and addresses a real toolchain limitation. The suggestions above are optional improvements to make the intent clearer and avoid potential CI issues with coverage reporting.

The core fix is good to merge as-is if you're comfortable with the test coverage gap for Xcode 15.x platforms.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.65%. Comparing base (34cb322) to head (e9ba884).

Additional details and impacted files

@@                Coverage Diff                 @@
##           107-swift-testing     #109   +/-   ##
==================================================
  Coverage              77.65%   77.65%           
==================================================
  Files                     74       74           
  Lines                   1553     1553           
==================================================
  Hits                    1206     1206           
  Misses                   347      347           

Flag	Coverage Δ
spm	76.94% <ø> (ø)
swift-	77.65% <ø> (ø)
ubuntu	77.65% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[claude]

PR Review: Build Configuration Changes for Xcode 15.x

Summary

This PR addresses Swift Testing macro resolution issues in Xcode 15.x by:

1. Removing Xcode 15.0.1 (Swift 5.9) from the build matrix
2. Adding build-only mode for Xcode 15.4 platform builds (iOS, watchOS)
3. Keeping Xcode 15.4 macOS builds running tests normally

Code Quality: ✅ Good

Strengths:

• Clean, focused changes targeting a specific technical issue
• Well-documented commit messages explaining the rationale
• Proper conditional logic using matrix.type != '' to distinguish platform builds from SPM builds

Potential Issues: ⚠️ Inconsistencies Found

1. Swift 5.9 Support Inconsistency

The PR removes Xcode 15.0.1 (Swift 5.9) from macOS builds, citing incompatibility with Swift Testing 0.5.1. However:

• Package.swift declares swift-tools-version:5.9 (line 1)
• Ubuntu build matrix still includes swift:5.9 container (line 27 of workflow)

Recommendation: For consistency, either update Package.swift to swift-tools-version:5.10 and remove Swift 5.9 from Ubuntu matrix, OR document why Swift 5.9 is supported on Ubuntu but not macOS.

2. Build-Only Mode Impact on Coverage

The build-only: true flag for Xcode 15.4 platform builds means these configurations won't run tests or generate coverage data. This is acceptable for compatibility, but consider documenting that Xcode 15.4 platform builds provide compilation validation only.

Performance Considerations: ✅ Good

• Reducing build matrix by removing 3 Xcode 15.0.1 entries improves CI runtime
• Build-only mode is faster than full test runs

Security Concerns: ✅ None Identified

• No security-related changes
• Workflow uses pinned action versions

Test Coverage: ⚠️ Reduced but Acceptable

Impact:

• Xcode 15.4 iOS and watchOS builds now skip tests
• Still have comprehensive coverage from Ubuntu, macOS Xcode 15.4 SPM builds, Xcode 16.4+, Windows and Android builds

Verdict: The reduction is justified given the macro plugin issues and redundant coverage from newer Xcode versions.

Additional Observations

1. Duplicate Matrix Entries

The workflow has duplicate Xcode 15.4 entries at lines 58-60 (SPM build) and lines 78-81 (macOS platform build). Consider consolidating or clarifying the difference in comments.

2. Conditional Logic Clarity

The condition contains(matrix.xcode, 'Xcode_15.4') && matrix.type != '' is correct but could benefit from a comment explaining it targets only platform builds.

Recommendations Summary

1. HIGH PRIORITY: Address Swift 5.9 inconsistency between Package.swift, macOS matrix, and Ubuntu matrix
2. MEDIUM PRIORITY: Add comments documenting why Xcode 15.4 platform builds use build-only mode
3. LOW PRIORITY: Clarify or consolidate duplicate Xcode 15.4 matrix entries
4. LOW PRIORITY: Add inline comments explaining the conditional logic for build-only mode

Verdict: ✅ Approve with Recommendations

The changes effectively solve the stated problem. The main concern is the Swift 5.9 support inconsistency, which should be addressed for clarity. Otherwise, the PR is well-executed and focused.

[claude]

PR Review: Add build-only mode for Xcode 15.x

Overview

This PR addresses Swift Testing macro resolution errors in Xcode 15.x by implementing a build-only mode for platform-specific tests while maintaining build validation. The changes are well-targeted and align with the project's CI/CD strategy.

---

Strengths

1. Pragmatic Solution: The build-only mode for Xcode 15.4 platform builds is a sensible approach that maintains CI coverage without blocking on known tooling limitations.

2. Clean Matrix Simplification: Removing Xcode 15.0.1 reduces CI complexity and runtime while keeping Xcode 15.4 for build validation.

3. Consistent Conditional Logic: The updated conditions properly target only Xcode 15.4 with precise matching and correctly excludes SPM-only builds from build-only mode.

4. Preserved Test Coverage: Full test execution still runs on Xcode 16.4+ and Ubuntu, maintaining comprehensive coverage where tooling supports it.

---

Issues and Recommendations

1. Package.swift Deletion - Needs Clarification

The PR deletes Package.swift which contained the package manifest for Swift 5.9+. This is concerning because there is no explanation in the PR description for why this file is being removed. If this is the primary package manifest, deletion would break SPM builds. The deleted manifest looked legitimate with proper dependencies.

Action Required: Please clarify why Package.swift is deleted, or restore it if this was unintentional.

2. Macro Fingerprint Validation Logic

The condition on line 233 now only applies when matrix.type is not empty, which excludes SPM builds. Question: Should SPM-only builds on Xcode 15.4 also need macro fingerprint validation disabled?

3. Build-Only Mode Coverage Gap

Platform-specific builds on Xcode 15.4 won't run tests (iOS 17.5, watchOS 10.5 missing coverage). This appears acceptable since tests run on newer OS versions and SPM builds on Xcode 15.4 should still run tests.

Recommendation: Confirm that SPM tests on Xcode 15.4 are still running.

---

Verification Checklist

Before merging, please confirm:

• Package.swift deletion is intentional and explained
• SPM builds on Xcode 15.4 still run full tests
• Xcode 16.4+ provides adequate test coverage for all platforms
• The build-only parameter is supported in brightdigit/swift-build@v1.4.2

---

Test Coverage Analysis

Maintained Full Testing: All Ubuntu builds, Windows, Android, Xcode 16.4+, and Xcode 15.4 SPM builds

Build-Only (No Tests): Xcode 15.4 platform-specific builds (iOS 17.5, watchOS 10.5, macOS)

This trade-off appears reasonable given the tooling constraints.

---

Code Quality: Good

Follows GitHub Actions best practices with clear conditional logic.

Security: No Concerns

Performance: Improved

Removing Xcode 15.0.1 entries (6 matrix jobs) will reduce CI time and costs.

---

Final Recommendation

Approve with clarification needed on the Package.swift deletion. The workflow changes look solid, but the package manifest removal needs explanation before merging.

---

Review generated with assistance from Claude Code