plugin validation with xcode build

[JaapWijnen]

I'm running into issues on CI where I'm building my code for iOS and have to use the xcodebuild command which fails with the following error.

error: the following command failed with exit code 1 but produced no further output
Validate plug-in “BenchmarkPlugin” in package “package-benchmark”

Any others that have dealt with this? I unfortunately can't add a OS condition to a plugin declaration in the Package.swift file.

[hassila]

Hey, sorry can't respond until mid august due to vacation time for everyone that could have input - hope someone else may chime in before.

[hassila]

Couldn't find anyone who recognised it, how does it work out of CI if you build from terminal?

[hassila]

Please reopen if you have further information.

[simlay]

I think I have run into a similar (or same) issue that the original author of this issue had.

Over in liveview-native/liveview-native-core#104, I've added benchmarks via this swift plugin. I must complement on this project. It's easy to write some benchmarks and simple to use.

Anyway, in my case, I've got some swift tests that normally work via xcodebuild -scheme LiveViewNativeCore -destination 'name=iPhone 15' test (it's called a bit deep in a cargo-make Makefile.toml to make sure the swift bindings and xcframework are generated from the rust).

When I add the following to my Package.swift:

        .executableTarget(
            name: "LiveViewNativeCoreBenchmarks",
            dependencies: [
                .product(name: "Benchmark", package: "package-benchmark"),
                "LiveViewNativeCore",
            ],
            path: "./crates/core/liveview-native-core-swift/Benchmarks/LiveViewNativeCore",
            plugins: [
                .plugin(name: "BenchmarkPlugin", package: "package-benchmark")
            ]
            ),

The xcodebuild command errors:

$ xcodebuild -scheme LiveViewNativeCore -destination 'name=iPhone 15' test
Command line invocation:
    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -scheme LiveViewNativeCore -destination "name=iPhone 15" test

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

Resolve Package Graph


Resolved source packages:
  swift-system: https://github.com/apple/swift-system @ 1.2.1
  Benchmark: https://github.com/ordo-one/package-benchmark @ 1.23.1
  SwiftDocCPlugin: https://github.com/apple/swift-docc-plugin @ 1.3.0
  TextTable: https://github.com/ordo-one/TextTable @ 0.0.2
  SymbolKit: https://github.com/apple/swift-docc-symbolkit @ 1.0.0
  jemalloc: https://github.com/ordo-one/package-jemalloc @ 1.0.0
  swift-atomics: https://github.com/apple/swift-atomics @ 1.2.0
  swift-argument-parser: https://github.com/apple/swift-argument-parser @ 1.3.1
  LiveViewNativeCore: /Users/sebastian.imlay/projects/liveview-native-core
  swift-numerics: https://github.com/apple/swift-numerics @ 1.0.2
  package-histogram: https://github.com/HdrHistogram/hdrhistogram-swift @ 0.1.2

2024-05-20 12:50:24.415 xcodebuild[17809:4526785] Writing error result bundle to /var/folders/b7/sqydsw156vxg0pk1nx9vqt600000gq/T/ResultBundle_2024-20-05_12-50-0024.xcresult
xcodebuild: error: Scheme LiveViewNativeCore is not currently configured for the test action.

Weirdly, swift test still works and run the tests that xcodebuild -scheme LiveViewNativeCore -destination 'name=iPhone 15' test would so, I have a hunch that this has nothing to do with the benchmarking plugin.

[simlay]

Well, I (mostly) figured it out. I'm not sure why but adding the executableTarget causes the xcodebuild scheme to change name and be suffixed with -Package. Then once I tried this I did run into a "Plugin validation" error. I found that someone had this issue with swiftlint and so adding in a -skipPackagePluginValidation fixes xcodebuild ... test fixes certain errors.

Definitely out of the scope of this issue but I then get various linktime/build errors:

• my iOS build says cannot find 'getRusage' in scope
• tvOS: Undefined symbol: for _mallctlbymib, _mallctlnametomib, _malloc_stats_print
• watchOS: ` No such module 'CDarwinOperatingSystemStats'

[hassila]

Hey @simlay

Well, I (mostly) figured it out. I'm not sure why but adding the executableTarget causes the xcodebuild scheme to change name and be suffixed with -Package. Then once I tried this I did run into a "Plugin validation" error. I found that someone had this issue with swiftlint and so adding in a -skipPackagePluginValidation fixes xcodebuild ... test fixes certain errors.

Definitely out of the scope of this issue but I then get various linktime/build errors:

• my iOS build says cannot find 'getRusage' in scope
• tvOS: Undefined symbol: for _mallctlbymib, _mallctlnametomib, _malloc_stats_print
• watchOS: ` No such module 'CDarwinOperatingSystemStats'

Yeah, a bit out of scope for this issue - we've only been running on macOS and Linux really, so I'm not surprised if there's some friction for other Apple platforms - for tvOS you might try disabling the jemalloc support as mentioned here:
https://swiftpackageindex.com/ordo-one/package-benchmark/1.22.4/documentation/benchmark/runningbenchmarks#Running-benchmarks-in-Xcode-and-using-Instruments-for-profiling-benchmarks

for iOS, there is an ifdef in OperatingSystemStatsProducer+Darwin.swift:

        #if os(macOS)

that makes getRusage to be unavailable - happy to take a PR on that ifdef (and any others if there's more) needed to get it to work on iOS (we currently don't build any benchmarks there internally).

Please open separate issues for any follow up that is concretely actionable if you don't mind?

[hassila]

@simlay I added the ifdefs here:
https://github.com/ordo-one/package-benchmark/releases/tag/1.23.4

Hopefully that makes it work on iOS at least, do give it a try.