[RFC] Standard interface for defining and consuming third-party dependencies

[pepicrft]

Need/problem

There are three dependency managers in the industry for Xcode projects: Carthage, CocoaPods, & Swift Package Manager.

• Swift Package Manager: Decentralized resolution, Xcode integrates the source code of dependencies into the build process.
• Carthage: Also decentralised resolution, but in this case the tool outputs fat dynamic frameworks that need to be manually added to the project.
• CocoaPods: Centralized resolution (Pods registry) and automatic integration using Xcode workspaces and xcconfig files.

Although Tuist provides an interface to integrate with them by describing their dependencies as target dependencies, the approach has proven to be problematic. Here's a list of issues that users reported:

• The generated workspaces might not contain the CocoaPods's Pods.xcodeproj project and therefore the linking of Pods fails at build times. Some teams have mitigated this issue by running some post-generation scripts that run pod install to integrate dependencies into the right workspace.
• Tuist doesn't have a knowledge on package transitive dependencies and therefore it does not know about dynamic libraries that might need to be embedded into final products. The integration only works if the packages are static libraries.
• If a Carthage framework has a transitive dependency, in essence another fat dynamic framework, developers have to add both as direct dependencies of a targets. Otherwise, Tuist won't embed the transitive dependencies into runnable products.

All of this results in a bad user experience and confused users. Moreover, with more people in the industry adopting Swift Package Manager as their dependency manager, projects are starting to feel the need of doing the same and have no clear path for transitioning.

Last but not least, the fact that CocoaPods and Carthage resolve the dependencies dynamically after the project has been generated, complicates the usage of caching because it's build on the assumptions that:

• The in-memory graph contains the full representation of the graph (including third-party dependencies).
• All nodes are cacheable, either because they are a fat binary or an xcframework, or because they can be turned into a cacheable xcframework.

Motivation

• Support all existing package managers with a standard interface. That'd give projects the flexibility to choose dependencies through any of the existing solutions.
• We'd provide a better experience integrating third-party dependencies because we'd be able to integrate a remote dependency-tree into the local dependency tree that represents users' projects.
• Allow caching modules that depend on third-party dependencies that are not Carthage frameworks.
• It's one more reason to adopt Tuist over other project generators.
• By integrating third-party dependencies as binaries teams would save some compilation time.

Detailed design

We'd standardise the contract between the dependency managers and Tuist. The contract would be:

• A Dependencies.resolved TOML file that describes the dependency tree.
• Dependencies as pre-compiled binaries. Each dependency would be either a xcframework or a fat dynamic binary, and support files like swift module interfaces or Objective-C headers.

Tuist/Dependencies.swift

We'd introduce a new manifest file that projects could use to list the third-party dependencies of their project:

import ProjectDescription

let dependencies = [
	.carthage("Alamofire/Alamofire", .branch("master")),
	.cocoapods("RxSwift", .exact("3.2.1")),
	.package("XcodeProj", "https://github.com/tuist/xcodeproj", .exact("3.2.1"))
]

tuist dependencies fetch/update

We'd add a new set of commands for pulling and updating dependencies. In the case of CocoaPods and SPM, it'd generate an Xcode project and then use it to build .xcframeworks. .xcframeworks would be stored under Tuist/Dependencies. Each subdirectory under that directory represents a dependency and contains in it both the binary, and the support files. The structure of the dependency's directory could be something along the lines of:

Tuist/
	Dependencies/
		Alamofire/
			Support/
			Alamofire.xcramework

Dependencies.resolved

Alongside the pre-compiled binaries for dependencies, Tuist would output a new file that lists the versions dependencies have been resolved to, their references, and the dependency tree.

New TargetDependency case

And last but not least, we'd add a new type of target dependency, .thirdParty, whose only value is the reference of the third-party dependency as it's shown in the Dependencies.resolved file:

Let target = Target(dependencies: [
  .thirdParty("Alamofire"), // Carthage
  .thirdParty("RxSwift"), // CocoaPods
  .thirdParty("XcodeProj"), // SPM
])

As part of building the in-memory graph, Tuist would use the information from the Dependencies.resolved to replace .thirdParty with the right dependency. This could be done from a graph mappers. For example, the example above would become:

let target = Target(dependencies: [
  .framework("//Tuist/Dependencies/Alamofire/Alamofire.framework"),
  .xcframework("//Tuist/Dependencies/RxSwift/RxSwift.xcframework),
  .xcframework("//Tuist/Dependencies/XcodeProj/XcodeProj.xcframework"),
])

Drawbacks

• We are creating a stronger contract between Tuist and the dependency managers and that increases the maintenance cost. Also the logic for turning Pods into .xcframeworks would most likely be written in Ruby (CocoaPods plugin) and contributors/maintainers might not be that familiar with writing and maintaining Ruby code.
• New versions of the dependency managers might break Tuist's functionality so we need to have a suite of tests that tests this functionality against the supported range of dependency manager versions.
• Since we'd integrate binaries, developers could debug the code of third-party dependencies by editing the code directly. This is not a big deal though because they could do the debugging using the project of those dependencies.

Alternatives

• Don't build such integration with the dependency managers and add a post-generation hook that developers can use to modify the projects generated by Tuist.

Adoption strategy

• Deprecation logs would point developers to the documentation of the new interfaces. The documentation page would include a migration guideline.

How we teach this

• We already have documentation about how to deal with third-party dependencies so we'd need to update it to onboard existing and new users into the new process.
• We could also write a post announcing the that we are standardising the consumption of third-party dependencies and point people to the documentation page.
• We could add deprecation logs to nudge developers to use the new APIs.

Unresolved questions

• Is TOML a good format for the Dependencies.resolved file?
  • Should we name it Dependencies.toml so that editors can apply code-highlighting automatically?
• Is Tuist/Dependencies the right directory for placing the binaries?
• Unlike Carthage & SPM, Pods had attributes to be applied to the projects that integrate them. Do we want Tuist to do that too?
• Is thirdParty a good name for referencing those dependencies or should we use something like .remote instead?
• What should we do in case developers do tuist generate and their projects reference a third-party dependency that have not been resolved by running tuist dependencies fetch?
• Should the integration with the SPM to build packages as .xcframeworks be through the CLI or should we link SPM and use the APIs directly?

[garvankeeley]

Carthage ... the tool outputs fat dynamic frameworks

Minor note: Firefox iOS uses it for dynamic frameworks, static frameworks, and pure source files (rather than also using git submodules).

[fortmarek]

Great RFC, @pepibumur 👏

This would be a great addition as it'd make it easier to combine various dependency managers and all the dependencies would be defined in manifests (which is great and makes it more comprehensible).

Now, let me go through a few points that came up to my mind while reading this:

• if we integrate Carthage, some teams might want to still keep their Rome integration as a lot of mid to large companies use it. Is there a story for that? Or is that something that could be resolved by tuist cache at some point?

• Is thirdParty a good name for referencing those dependencies or should we use something like .remote instead?

.remote might not be the right name - after all you can import local packages and in that case the naming would be confusing. Although, .thirdParty also doesn't seem really right since you can import your own library, but it's still better than .remote IMHO. I thought about simple .dependency but might be confusing with others, too, so I'm probably in favor of .thirdParty

• How do you plan to generate .toml file? Do we introduce our own logic of resolving dependencies instead of relying on the managers themselves? It'd also mean discarding .package.resolved, so that's just something to think about and add it into migration guideline.

• Is TOML a good format for the Dependencies.resolved file?

It's the first time I am hearing about it, but if it's something that is used why not. A lot of dependency managers use their own format, so it'd be nice to use some standard and leverage code-highlighting with .toml extension

• Is Tuist/Dependencies the right directory for placing the binaries?

Shouldn't it be in Derived directory maybe? 🤔

• Unlike Carthage & SPM, Pods had attributes to be applied to the projects that integrate them. Do we want Tuist to do that too?

I think this can be omitted in the first iteration and we can add if we see the demand

• What should we do in case developers do tuist generate and their projects reference a third-party dependency that have not been resolved by running tuist dependencies fetch?

Maybe fetch can be part of tuist generate? It's the case already for SPM and cocoapods, so it'd make sense to translate it to this feature, too.

• Should the integration with the SPM to build packages as .xcframeworks be through the CLI or should we link SPM and use the APIs directly?

From the codebase-perspective, it'd be nice to use the APIs directly, but it might create some confusion for the users who might expect that the build is done with their currently selected toolchain, whereas if we used APIs directly, it wouldn't reflect that.

Overall, I am in favor of this change, but it's a trivial feature. My suggestion is to try to use the proposed logic for SPM where we already have tight control and so it might be the easiest manager to port - and while our current solution for SPM is good, we still delegate a lot to Xcode and that means there are things we are not able to optimize for. For example one thing we could do if we have more control, is to more efficiently resolve dependencies when running tuist focus because we'll need to go back where we fetch dependencies anew for every subproject as per [this issue]. This is memory-intensive and can take quite long, but with the proposed approach we probably would not have to fetch any new dependencies and just take them from .toml file and dependencies folder

[pepicrft]

if we integrate Carthage, some teams might want to still keep their Rome integration as a lot of mid to large companies use it. Is there a story for that? Or is that something that could be resolved by tuist cache at some point?

In combination with the cache, developers won't have to use Rome anymore. I know this means moving away from the setup that they already invested in, but they'll benefit from having caching at all the levels, including for their local modules.

.remote might not be the right name - after all you can import local packages and in that case the naming would be confusing. Although, .thirdParty also doesn't seem really right since you can import your own library, but it's still better than .remote IMHO. I thought about simple .dependency but might be confusing with others, too, so I'm probably in favor of .thirdParty

We can call it .remote too. That's the other option I was thinking about. It feels odd that you are pulling a dependency of your own from another repository and you call it "third-party".

How do you plan to generate .toml file? Do we introduce our own logic of resolving dependencies instead of relying on the managers themselves? It'd also mean discarding .package.resolved, so that's just something to think about and add it into migration guideline.

There are libraries that we can use for that. The resolution of dependencies will be done by the package managers so I think we can keep their lockfiles as the source of truth:

Dependencies/
  Dependencies.toml
  Podfile.lock
  Cartfile.lock
  Package.resolved

It's the first time I am hearing about it, but if it's something that is used why not. A lot of dependency managers use their own format, so it'd be nice to use some standard and leverage code-highlighting with .toml extension

I'll in fact play with JSON, YAML, and TOML to see which one outputs a more human-readable format. I think it's great being able to open that file and understanding it at a glance without having to make an effort to parse it. For example, if we end up with many indentations, YAML might be hard for people to process. It's hard to say though until we know more about the structure of this file.

Shouldn't it be in Derived directory maybe? 🤔

I suggested Tuist because dependencies will be global, and that's what the Tuist directory is for. The Derived directory is generated alongisde each project. Alternatively, we could create a global Derived directory but that's what the Tuist directory is for.

I think this can be omitted in the first iteration and we can add if we see the demand

Agree

Maybe fetch can be part of tuist generate? It's the case already for SPM and cocoapods, so it'd make sense to translate it to this feature, too.

SPM does it. I think for the first iteration we can keep them separated, and once we validate this idea, we can think about integrating it into generate. SPM does it, so it's a behavior that developers are already getting used to.

Overall, I am in favor of this change, but it's a trivial feature. My suggestion is to try to use the proposed logic for SPM where we already have tight control and so it might be the easiest manager to port - and while our current solution for SPM is good, we still delegate a lot to Xcode and that means there are things we are not able to optimize for. For example one thing we could do if we have more control, is to more efficiently resolve dependencies when running tuist focus because we'll need to go back where we fetch dependencies anew for every subproject as per [this issue]. This is memory-intensive and can take quite long, but with the proposed approach we probably would not have to fetch any new dependencies and just take them from .toml file and dependencies folder

I like the idea of starting with a dependency manager. I'd do Carthage though because it already provides the dependencies in the format that we expect: pre-compiled binaries. If that works, we can then do the SPM because it's easier to integrate with our Swift codebase. And last, I'd do CocoaPods because it's in Ruby and I think the integration will be more complex.

[pepicrft]

This has already been done and therefore can be closed