.package.resolved is not generated

[tdkn]

What happened?

If you use Local Package with Xcode's default integration mechanism, the .package.resolved file is not generated by Tuist, so builds will fail on Xcode Cloud, which expects the existence of a *.xcworkspace/xcshareddata/swiftpm/Package.resolved file.

How do we reproduce it?

1. Clone https://github.com/tdkn/tuist-minimal-reproduction (or zipped version is here)
2. Run tuist generate --no-open
3. The .package.resolved file should be present but is missing.
4. Also, *.xcworkspace/xcshareddata/swiftpm/Package.resolved should have been generated, but it's not there.
5. Simulate Xcode Cloud's Resolve package dependencies task locally.
6. And then an error occurs.

Error log

Simulate Xcode Cloud's Resolve package dependencies task locally.

$ xcodebuild -resolvePackageDependencies -onlyUsePackageVersionsFromResolvedFile -disableAutomaticPackageResolution -workspace ./MyWorkspace.xcworkspace -scheme TuistMinimalReproduction -hideShellScriptEnvironment

The following error outputs:

Command line invocation:
    /Applications/Xcode-15.4.0.app/Contents/Developer/usr/bin/xcodebuild -resolvePackageDependencies -onlyUsePackageVersionsFromResolvedFile -disableAutomaticPackageResolution -workspace ./MyWorkspace.xcworkspace -scheme TuistMinimalReproduction -hideShellScriptEnvironment

User defaults from command line:
    HideShellScriptEnvironment = YES
    IDEPackageOnlyUseVersionsFromResolvedFile = YES
    IDEPackageSupportUseBuiltinSCM = YES

Resolve Package Graph
a resolved file is required when automatic dependency resolution is disabled and should be placed at /Users/tdkn/Workspaces/private/TuistMinimalReproduction/MyWorkspace.xcworkspace/xcshareddata/swiftpm/Package.resolved. Running resolver because the following dependencies were added: 'swift-algorithms' (https://github.com/apple/swift-algorithms)

Resolved source packages:
  MyPackage: /Users/tdkn/Workspaces/private/TuistMinimalReproduction/MyPackage

2024-05-28 16:08:50.337 xcodebuild[90624:1899640] Writing error result bundle to /var/folders/bz/2q5vvytd059by91lfx4b39380000gn/T/ResultBundle_2024-28-05_16-08-0050.xcresult
xcodebuild: error: Could not resolve package dependencies:
  a resolved file is required when automatic dependency resolution is disabled and should be placed at /Users/tdkn/Workspaces/private/TuistMinimalReproduction/MyWorkspace.xcworkspace/xcshareddata/swiftpm/Package.resolved. Running resolver because the following dependencies were added: 'swift-algorithms' (https://github.com/apple/swift-algorithms)

Screenshot of Xcode Cloud Log (Web)

macOS version

14.4.1

Tuist version

4.14.0

Xcode version

15.4

[Lilfaen]

Copied from Slack:

If I run tuist generate while Xcode is closed, everything works as expected.
If I run tuist generate while the Xcode Project is open and/or focused, I get the following error:

We received an error that we couldn't handle:
    - Localized description: The file "Package.resolved" couldn't be opened because there is no such file.
    - Error: Error Domain=NSCocoaErrorDomain Code=260 "The file "Package.resolved" couldn't be opened because there is no such file." UserInfo={NSFilePath=/Users/clemensbeck/repro/Project.xcworkspace/xcshareddata/swiftpm/Package.resolved, NSUnderlyingError=0x600000dcc990 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
If you think it's a legit issue, please file an issue including the reproducible steps: https://github.com/tuist/tuist/issues/new/choose
Consider creating an issue using the following link: https://github.com/tuist/tuist/issues/new/choose

Since the .package.resolved file is a copy of the Package.resolved file, which can't be found, it gets deleted.
It happens sometimes if Xcode is open. But it happens most, almost all of the times if Xcode is the focused window.
This is quite difficult to reproduce, for projects that take longer to generate it is more probable to happen, since the dev then switches back to Xcode while it's generating.
I attached a small example project for reproduction, is anyone else able to reproduce this? 🫠 You might have to quickly switch to Xcode after running tuist generate

Here's the example project:
Project.zip

[pepicrft]

For anyone interested in looking into this one, Tuist copies the Package.resolved in and out from the generated .xcodeproj, since teams typically Git-ignore it, and that'd cause non-deterministic dependency resolution.
Based on the errors, there seems to be a race condition between:

• Tuist trying to copy it outside of the Xcode project
• Xcode re-creating the file while resolving the dependencies.

I'd investigate if there's a way to control where the file gets generated, so that we prevent the race condition.

[tdkn]

My original problem and Lilfaen's problem (comment #2) appear to have different causes and should be considered as separate issues.

Now, I cloned Tusit locally and tried to debug it.
Line at SwiftPackageManagerInteractor.swift:51, graphTraverser.hasRemotePackages became false and the step where .package.resolved is generated was skipped.

tuist/Sources/TuistGenerator/Utils/SwiftPackageManagerInteractor.swift

Lines 44 to 59 in 02b882a

	private func generatePackageDependencyManager(
	at path: AbsolutePath,
	workspaceName: String,
	config: Config,
	graphTraverser: GraphTraversing
	) async throws {
	guard !config.generationOptions.disablePackageVersionLocking,
	graphTraverser.hasRemotePackages
	else {
	return
	}
	let rootPackageResolvedPath = path.appending(component: ".package.resolved")
	let workspacePackageResolvedFolderPath = path
	.appending(try RelativePath(validating: "\(workspaceName)/xcshareddata/swiftpm"))
	let workspacePackageResolvedPath = workspacePackageResolvedFolderPath.appending(component: "Package.resolved")

In my minimal reproduction repository, MyPackage/Package.swift depends on an external package called swift-algorithms, so I think graphTraverser.hasRemotePackages needs to be true.

https://github.com/tdkn/tuist-minimal-reproduction/blob/03a87ef280ddff011fadc5b3e8a84f8c4340c184/MyPackage/Package.swift#L19

[pepicrft]

@tdkn how are you integrating those packages? If you are referencing those as .external, then graphTraverser.hasRemotePackages returns false because the function only considers remote packages, that are integrated using Xcode's standard integration (the name is a bit misleading). In that case, there's no lockfile managed by Xcode, and therefore, it makes sense that the logic is being skipped.
The issue should only be reproducible if dependencies are integrated using Xcode's integration.

[tdkn]

@pepicrft
As mentioned in Project.swift, the main target depends on MyPackage and MyPackage is configured to depend on swift-algorithms.

.external is not used.

graph TD;
    main_target["main target (TuistMinimalReproduction)"] --> MyPackage;
    MyPackage --> swift-algorithms["swift-algorithms"];
    swift-algorithms["swift-algorithms"] --> swift-numerics["swift-numerics"]

Loading

// Project.swift

let project = Project(
    name: "TuistMinimalReproduction",
    packages: [
        .package(path: "MyPackage"),
    ],
    targets: [
        .target(
            name: "TuistMinimalReproduction",
            /* ... */
            dependencies: [
                .package(product: "MyPackage", type: .runtime),
            ]
        ),
    ]
)

// MyPackage/Package.swift

let package = Package(
    name: "MyPackage",
    platforms: [.macOS(.v13), .iOS(.v16)],
    products: [
        .library(name: "MyPackage", targets: ["MyPackage"]),
    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-algorithms", .upToNextMajor(from: "1.2.0")),
    ],
    targets: [
        .target(
            name: "MyPackage",
            dependencies: [
                .product(name: "Algorithms", package: "swift-algorithms"),
            ]
        ),
    ]
)

When you open the workspace with tuist generate, Xcode will resolve the dependencies, as shown in the attached screenshot below. It then writes it to MyWorkspace.xcworkspace/xcshareddata/swiftpm/Package.resolved.

So tuist should generate .package.resolved in the same way.

[pepicrft]

Thanks @tdkn 🙏🏼. That's, in fact, what we do; after we generate the project, we call an xcodebuild command to resolve dependencies, and when it completes, we assume the resolved file is present in the Xcode project. However, based on this bug, completing the command doesn't necessarily mean that the lockfile is present, which would explain why the error happens.

[tdkn]

@pepicrft

after we generate the project, we call an xcodebuild command to resolve dependencies

Hmm, in this issue situation, I don't think xcodebuild --resolvePackageDependencies is called, am I wrong?
Do you think the original issue was caused by the race condition?

In this reproduction, I believe that xcodebuild --resolvePackageDependencies is never executed by Tuist because it is always returned on line SwiftPackageManagerInteractor.swift:53.

And if I intentionally comment out line SwiftPackageManagerInteractor.swift:51, .package.resolved is generated as it goes through the process of running xcodebuild.

[woin2ee]

I think this issue happens when a project depends on a local swift package, and that local swift package depends on remote packages.

In this case, if the generated project is opened, then Xcode resolves remote packages and create .package.resolved file into the workspace. However, the project is not opened, .package.resolved is never created because graphTraverser.hasRemotePackages is false.(project has only local package directly)

In conclusion, this issue is associated with resolving package dependencies in virtual environment such as Xcode Cloud.

Am I right about this? @tdkn

[pepicrft]

Thanks for the thorough analysis @woin2ee. What you describes makes total sense as potential cause of the issue.
I wonder if this PR that I merged a couple of weeks ago mitigates the issue. Could you check @tdkn ?

[tdkn]

@woin2ee
First of all, thank you for your interest in this issue. I answer your question.

In this case, if the generated project is opened, then Xcode resolves remote packages and create .package.resolved file into the workspace. However, the project is not opened, .package.resolved is never created because graphTraverser.hasRemotePackages is false.(project has only local package directly)

No, to be precise, opening a workspace does not generate .package.resolved. What is generated when you open a workspace is *.xcworkspace/xcshareddata/swiftpm/Package.resolved, not .package.resolved.

I don't fully understand the logic of graphTraverser, but in situations like my minimal reproduction repo, I think that graphTraverser.hasRemotePackages should be true because it indirectly depends on the remote package, and xcodebuild --resolvedPackageDependencies should be executed by System.shared.run (SwiftPackageManagerInteractor.swift#L87).

@pepicrft
I tried both v4.18.0 and the latest v4.19.0, but the issue where .package.resolved is not generated was not resolved 😢

[Udobnaja]

@pepicrft
Hey!
I noticed that this issue is tagged as 'good first issue'. I'd like to try it.

But first, I'd like to ask: since it's good for newcomers, do you already have some ideas or directions for solving it?
For example, replacing hasRemotePackages with something like hasPackages (always applying logic if there are packages) or maybe changing the logic for forming graphTraverser and merging packages there?

[pepicrft]

I don't fully understand the logic of graphTraverser, but in situations like my minimal reproduction repo, I think that graphTraverser.hasRemotePackages should be true because it indirectly depends on the remote package, and xcodebuild --resolvedPackageDependencies should be executed by System.shared.run (SwiftPackageManagerInteractor.swift#L87).

Thanks a lot @Udobnaja for your interest in contributing to this one :). I think @tdkn's suggestion makes total sense. We can adjust the logic of the mentioned function to return true, and make sure we have an acceptance test that covers the scenario to prevent regressions in the future.