Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage #8894

simonjbeaumont · 2025-07-04T09:42:47Z

Motivation

The current setting for LLVM_PROFILE_PATH, used for code coverage, leads to corrupt profile data when tests are run in parallel or when writing "exit tests" with Swift Testing. This also results in the swift test --enable-code-coverage command to fail.

The LLVM_PROFILE_PATH environment variable is used by the runtime to write raw profile files, which are then processed when the test command finishes to produce the coverage results as JSON. The variable supports several pattern variables¹, including %Nm, which is currently set, and is documented to create a pool of files that the runtime will handle synchronisation of. This is fine for parallelism within the process but will not work across different processes. SwiftPM uses multiple invocations of the same binary for parallel testing and users may also fork processes within their tests, which is now a required workflow when using exit tests with Swift Testing, which will fork the process internally. Furthermore, the current setting for this variable uses only %m (which implies N=1), which makes it even more likely that processes will stomp over each other when writing the raw profile data.

We can see a discussion of this happening in practice in #8893.

The variable also supports %p¹, which will expand to produce a per-process path for the raw profile, which is probably what we want here, since Swift PM is combining all the profiles in the configured directory.

Modifications

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage.

Result

Running tests write coverage raw profile data to their own per-process file pool.
Running tests in parallel with code coverage no longer risks corrupting coverage data.
Running exit tests no longer risks corrupting coverage data.
Fixes Multiple exit tests result in corrupt coverage profdata, which leads to test command failure, even with .serialized #8893.

Appendix: Demonstrating the merging of per-process profiles

// file: ReproTests.swift

import Testing
import struct Foundation.URL
#if canImport(Darwin)
import Darwin
#elseif canImport(Glibc)
import Glibc
#endif

@Suite(.serialized) struct Suite {
    static func updateLLVMProfilePath() {
        let key = "LLVM_PROFILE_FILE"
        let profrawExtension = "profraw"
        guard let previousValueCString = getenv(key) else { return }
        let previousValue = String(cString: previousValueCString)
        let previousPath = URL(filePath: previousValue)
        guard previousPath.pathExtension == profrawExtension else { return }
        guard !previousPath.lastPathComponent.contains("%p") else { return }
        let newPath = previousPath.deletingPathExtension().appendingPathExtension("%p").appendingPathExtension(profrawExtension)
        let newValue = newPath.path(percentEncoded: false)
        print("Replacing \(key)=\(previousValue) with \(key)=\(newValue)")
        setenv(key, newValue, 1)
    }

    @Test func testA() async {
        Self.updateLLVMProfilePath()
        await #expect(processExitsWith: .success) { Subject.a() }
    }

    @Test func testB() async {
        Self.updateLLVMProfilePath()
        await #expect(processExitsWith: .success) { Subject.b() }
    }
}

// file: Subject.swift

struct Subject {
    static func a() { _ = "a" }
    static func b() { _ = "a" }
}

Running with just one test results in one per-process profile and 50% coverage, as expected.

% swift test --enable-code-coverage --filter Suite.testa
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testa() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testa() passed after 0.018 seconds.
✔ Suite Suite passed after 0.018 seconds.
✔ Test run with 1 test in 1 suite passed after 0.018 seconds.

% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15828.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 1,
  "percent": 50
}

Running the other test also results in one per-process profile and 50% coverage, as expected.

% swift test --enable-code-coverage --filter Suite.testb
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testb() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testb() passed after 0.017 seconds.
✔ Suite Suite passed after 0.017 seconds.
✔ Test run with 1 test in 1 suite passed after 0.017 seconds.

% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15905.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 1,
  "percent": 50
}

Running both tests results in two per-process profile and 100% coverage, after merge.

% swift test --enable-code-coverage --filter Suite.testa --filter Suite.testb
...
◇ Test run started.
↳ Testing Library Version: 6.2 (9ebfc4ebbb2840d)
↳ Target Platform: aarch64-unknown-linux-gnu
◇ Suite Suite started.
◇ Test testa() started.
Replacing LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.profraw with LLVM_PROFILE_FILE=/pwd/.build/aarch64-unknown-linux-gnu/debug/codecov/Swift Testing%m.%p.profraw
✔ Test testa() passed after 0.016 seconds.
◇ Test testb() started.
✔ Test testb() passed after 0.015 seconds.
✔ Suite Suite passed after 0.033 seconds.
✔ Test run with 2 tests in 1 suite passed after 0.033 seconds.

% ls -1 .build/debug/codecov/
default.profdata
repro-exit-tests-coverage-corruption.json
'Swift Testing12847901981426048528_0.15981.profraw'
'Swift Testing12847901981426048528_0.15988.profraw'
'Swift Testing12847901981426048528_0.profraw'
XCTest12847901981426048528_0.profraw

% cat .build/debug/codecov/repro-exit-tests-coverage-corruption.json  | jq '.data[].files[] | select(.filename == "/pwd/Tests/ReproTests/Subject.swift").summary.functions'
{
  "count": 2,
  "covered": 2,
  "percent": 100
}

https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#running-the-instrumented-program ↩ ↩²

dschaefer2 · 2025-07-04T19:32:44Z

@swift-ci please test

dschaefer2 · 2025-07-04T19:35:04Z

Thanks Si! Can you add some simple tests here to make sure the attributes in the path resolve correctly?

simonjbeaumont · 2025-07-04T21:21:58Z

@dschaefer2 sure thing. I couldn't see any tests that check for the coverage output, but I maybe missed them? Could you point me at them if they exist or, if they don't, suggest the best place to add the tests (don't super know my way around in SwiftPM).

The PR description shows the impact of running with %p with an out of tree workaround.

simonjbeaumont · 2025-07-07T09:35:37Z

@dschaefer2 I added a test in 5290b2b. I couldn't see anywhere where there were already tests for how the coverage files got exported, merged, or validated, so I decided to put them in integration tests.

dschaefer2 · 2025-07-07T14:26:46Z

@dschaefer2 I added a test in 5290b2b. I couldn't see anywhere where there were already tests for how the coverage files got exported, merged, or validated, so I decided to put them in integration tests.

Nice. Thanks!

dschaefer2 · 2025-07-07T14:26:57Z

@swift-ci please test

dschaefer2 · 2025-07-07T14:27:09Z

@swift-ci please test windows

simonjbeaumont · 2025-07-08T16:48:50Z

@dschaefer2 I notice the Linux jobs are still running with Swift 6.1, is that expected?

[2025-07-07T14:37:31.549Z] ◇ Test run started.
[2025-07-07T14:37:31.549Z] ↳ Testing Library Version: 6.1 (43b6f88e2f2712e)

— https://ci.swift.org/job/swift-package-manager-self-hosted-Linux-smoke-test/6515/consoleText

I will update the test to either skip on pre-6.2 (because it uses exit tests) or to try and validate some other way. But just checking that's what we expect.

I have validated locally that they do pass with the 6.2 nightly Docker image.

simonjbeaumont · 2025-07-08T16:52:22Z

@swift-ci please test

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage

0b3beec

simonjbeaumont requested review from jakepetroules, dschaefer2, shawnhyam, bripeticca, plemarquand, owenv, bkhouri, cmcgee1024 and daveyc123 as code owners July 4, 2025 09:42

dschaefer2 added the needs tests This change needs test coverage label Jul 4, 2025

Add integration test

5290b2b

simonjbeaumont added 2 commits July 8, 2025 15:23

fixup: Fix JSON schema and typo in comment

bca1638

Only run exit tests on 6.2+

3b48cda

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage #8894

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage #8894

simonjbeaumont commented Jul 4, 2025

Uh oh!

dschaefer2 commented Jul 4, 2025

Uh oh!

dschaefer2 commented Jul 4, 2025

Uh oh!

simonjbeaumont commented Jul 4, 2025 •

edited

Loading

Uh oh!

simonjbeaumont commented Jul 7, 2025

Uh oh!

dschaefer2 commented Jul 7, 2025

Uh oh!

dschaefer2 commented Jul 7, 2025

Uh oh!

dschaefer2 commented Jul 7, 2025

Uh oh!

simonjbeaumont commented Jul 8, 2025

Uh oh!

simonjbeaumont commented Jul 8, 2025

Uh oh!

Uh oh!

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage #8894

Are you sure you want to change the base?

Add %p to LLVM_PROFILE_FILE pattern when running tests with coverage #8894

Conversation

simonjbeaumont commented Jul 4, 2025

Motivation

Modifications

Result

Appendix: Demonstrating the merging of per-process profiles

Footnotes

Uh oh!

dschaefer2 commented Jul 4, 2025

Uh oh!

dschaefer2 commented Jul 4, 2025

Uh oh!

simonjbeaumont commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonjbeaumont commented Jul 7, 2025

Uh oh!

dschaefer2 commented Jul 7, 2025

Uh oh!

dschaefer2 commented Jul 7, 2025

Uh oh!

dschaefer2 commented Jul 7, 2025

Uh oh!

simonjbeaumont commented Jul 8, 2025

Uh oh!

simonjbeaumont commented Jul 8, 2025

Uh oh!

Uh oh!

simonjbeaumont commented Jul 4, 2025 •

edited

Loading