Xcframework prebuilt binaries #3123

igstewart3 · 2021-02-05T01:38:08Z

This PR adds the ability to retrieve XCFramework prebuilt binaries. It installs them into the Carthage/Build folder and skips any Xcode.swift steps that involve checking the architecture.

It could probably use a wee bit of guidance from someone more familiar with the project to ensure I haven't missed something important.

I have tested this on a local project and was successfully able to install XCFramework binaries, as well as regular Framework binaries from a previous release.

Note: I have not been able to run the unit tests yet due to some Swift compile errors I haven't been able to resolve.

sebskuse · 2021-02-09T08:52:01Z

I've tested this against the latest https://github.com/aws-amplify/aws-sdk-ios, which provides prebuilt XCFramework binaries and this works as expected!

Using Carthage 0.37.0 we see:

*** Fetching aws-sdk-ios
*** Checking out aws-sdk-ios at "2.22.2"
*** No cache found for aws-sdk-ios, building with all downstream dependencies
*** xcodebuild output can be found in /var/folders/c0/068z__x16p15190541prsvkm0000gp/T/carthage-xcodebuild.RnWKmY.log
*** Downloading aws-sdk-ios.framework binary at "AWS SDK for iOS 2.22.2"
***  Skipped installing aws-sdk-ios.framework binary due to the error:
	"Invalid archive - Found multiple frameworks with the same unarchiving destination:
* 	file:///var/folders/c0/068z__x16p15190541prsvkm0000gp/T/carthage-archive.IXODoQ/AWSLogs.xcframework/ios-arm64_i386_x86_64-simulator/AWSLogs.framework/
	file:///var/folders/c0/068z__x16p15190541prsvkm0000gp/T/carthage-archive.IXODoQ/AWSLogs.xcframework/ios-arm64_armv7/AWSLogs.framework/ 
		to:
	file:///Path/To/Carthage/Build/iOS/AWSLogs.framework/

... repeat for every framework

Falling back to building from the source

Using this branch we see:

*** Fetching aws-sdk-ios
*** Checking out aws-sdk-ios at "2.22.2"
*** No cache found for aws-sdk-ios, building with all downstream dependencies
*** xcodebuild output can be found in /var/folders/c0/068z__x16p15190541prsvkm0000gp/T/carthage-xcodebuild.ll5AoF.log
*** Downloading aws-sdk-ios.framework binary at "AWS SDK for iOS 2.22.2"

And the prebuilt frameworks are in the expected folder.

dmiluski · 2021-02-10T00:30:03Z

Hi @igstewart3 ,

I just ran into this today with a similar hiccup with AWS (wowza HUGE install times). Thanks for kicking off the effort.

I'm not a repo owner/maintainer, but wanted to give some notes as I just started dabbling with this too:

Reading through I noticed this used string literals and bool logic to concatenate framework download behaviors. Skimming the implementation, I noticed:

/// Describes the type of product built by an Xcode target.
public enum ProductType: String {
	/// A framework bundle.
	case framework = "com.apple.product-type.framework"

	/// A static library.
	case staticLibrary = "com.apple.product-type.library.static"

	/// A unit test bundle.
	case testBundle = "com.apple.product-type.bundle.unit-test"

And was wondering if perhaps xcframework should be included as a sibling here? Allowing for the use of this definition for path? Allowing for a model to drive potential differentiation rather than bool checks?

/// Framework Suffix
public enum FrameworkSuffix: String {

    /// Dynamic Framework
    case dynamic = "framework"

    /// XCFramework
    case xcode = "xcframework"
}

igstewart3 · 2021-02-15T03:48:52Z

Hi @igstewart3 ,

I just ran into this today with a similar hiccup with AWS (wowza HUGE install times). Thanks for kicking off the effort.

I'm not a repo owner/maintainer, but wanted to give some notes as I just started dabbling with this too:

Reading through I noticed this used string literals and bool logic to concatenate framework download behaviors. Skimming the implementation, I noticed:
/// Describes the type of product built by an Xcode target.
public enum ProductType: String {
	/// A framework bundle.
	case framework = "com.apple.product-type.framework"

	/// A static library.
	case staticLibrary = "com.apple.product-type.library.static"

	/// A unit test bundle.
	case testBundle = "com.apple.product-type.bundle.unit-test"
And was wondering if perhaps xcframework should be included as a sibling here? Allowing for the use of this definition for path? Allowing for a model to drive potential differentiation rather than bool checks?
/// Framework Suffix
public enum FrameworkSuffix: String {

    /// Dynamic Framework
    case dynamic = "framework"

    /// XCFramework
    case xcode = "xcframework"
}

Good shout, I'll take a look at tidying that up 👍

finik · 2021-02-23T06:29:36Z

I can confirm as well, the fix in this branch works, please merge and release this ASAP, this blocks some frameworks from being properly distributed. Thanks

ozgur · 2021-02-23T12:16:38Z

Source/CarthageKit/Project.swift

@@ -1447,11 +1457,13 @@ func platformForFramework(_ frameworkURL: URL) -> SignalProducer<SDK, CarthageEr
 /// Sends the URL to each framework bundle found in the given directory.
 internal func frameworksInDirectory(_ directoryURL: URL) -> SignalProducer<URL, CarthageError> {
 	return filesInDirectory(directoryURL, kUTTypeFramework as String)
+        .concat(filesInDirectory(directoryURL, "com.apple.xcframework"))


indentation fix needed.

ozgur · 2021-02-23T12:17:44Z

Source/CarthageKit/Xcode.swift

@@ -1413,6 +1419,9 @@ public func nonDestructivelyStripArchitectures(_ frameworkURL: URL, _ architectu

 /// Strips the given architectures from a framework.
 private func stripArchitectures(_ packageURL: URL, _ architectures: Set<String>) -> SignalProducer<(), CarthageError> {
+    guard isNotXCFramework(packageURL) else {


indentation fix needed. Please check the indentation of return statement below it.

ozgur · 2021-02-23T12:17:56Z

Source/CarthageKit/Xcode.swift

@@ -1431,6 +1440,9 @@ private func stripArchitectures(_ packageURL: URL, _ architectures: Set<String>)

 // Returns a signal of all architectures present in a given package.
 public func architecturesInPackage(_ packageURL: URL, xcrunQuery: [String] = ["lipo", "-info"]) -> SignalProducer<[String], CarthageError> {
+    guard isNotXCFramework(packageURL) else {


indentation fix needed. Please check the indentation of return statement below it.

Done for all changes made in the PR 👍

elliottwilliams

First off, thanks so much for your contribution! I think this makes a lot of sense and is obviously something we need to support.

Aside from the suggestions below, we need to handle checkFrameworkCompatibility correctly. Since it's possible to build an xcframework with -allow-internal-distribution, it's possible to have swiftmodules in the xcframework that aren't compatible with the machine they're downloaded to. This patch should fix it, LMK if there's anything I can clarify:
carthage-3123-additions.patch.txt

Source/CarthageKit/Project.swift

elliottwilliams · 2021-02-25T19:39:43Z

Source/CarthageKit/Xcode.swift

+
+private func isNotXCFramework(_ frameworkUrl: URL) -> Bool {
+	return FrameworkSuffix.from(string: frameworkUrl.pathExtension).value != .xcframework
+}


What is this (and all its call sites) doing? AFAICT this is related to copy-frameworks, not downloading binaries?

Several of these methods are called when installing the prebuilt XCFrameworks and fail due to not being able to find the executableURL in the bundle in binaryURL(packageUrl:). We could look into adding a custom case for XCFrameworks like is already in place for .dSYM and (and changing to return an array of URLs), but many of these processes (stripping architectures and debug symbols) don't necessarily make sense for an XCFramework anyway.

Source/CarthageKit/Project.swift

igstewart3 · 2021-03-01T02:00:38Z

@elliottwilliams Thanks very much for the feedback! I've implemented most of it, including the patch you provided. I think only the XCFramework guard cases in Xcode.swift still need to be resolved. They definitely allow the prebuilt binary to be installed successfully but they do feel a little clumsy, keen to hear your thoughts on that.

…orks

elliottwilliams · 2021-03-03T06:10:01Z

@elliottwilliams Thanks very much for the feedback! I've implemented most of it, including the patch you provided. I think only the XCFramework guard cases in Xcode.swift still need to be resolved. They definitely allow the prebuilt binary to be installed successfully but they do feel a little clumsy, keen to hear your thoughts on that.

@igstewart3 I think I got it working – we just need to avoid calling into the dSYM/bcsymbolmap logic higher up. This is similar to what we do on the build side, where we conditionally avoid calling any debug symbols merging code when making an xcframework. See the commit I just pushed :)

I confirmed this works with a github "aws-amplify/aws-sdk-ios" Cartfile, but LMK if there's anything else you're noticing.

igstewart3 · 2021-03-03T19:27:42Z

@elliottwilliams Nice! Good fix, that looks a lot tidier. Everything seems to be working in my tests as well 👍

elliottwilliams

Sounds good! Going to give the other maintainers a moment to review if they'd like.

… update --use-xcframeworks` when installing `dd-sdk-ios` in other app. We need this as `arm64` is disabled in PLCrashReporter simulator builds, so we cannot produce `DatadogCrashReporting.xcframework` as it is not able to link `CrashReporter` to our `ios-arm64_x86_64-simulator` slice. This change makes our SDK produce smaller slice, which is enough for running on simulator. This was discovered as Carthage doesn’t yet support pre-build xcframeworks: Carthage/Carthage#3123

Ian Stewart added 3 commits February 5, 2021 11:31

Allow XCFramework prebuilt binaries

11ee044

Ignore XCFramework binaries when doing architecture checks

1b939b0

Install XCFrameworks in root of Build

0c5ee02

dmiluski mentioned this pull request Feb 10, 2021

Invalid archive - Found multiple frameworks with the same unarchiving destination aws-amplify/aws-sdk-ios#3430

Closed

Ian Stewart added 2 commits February 17, 2021 16:18

Add FrameworkSuffix enum for framework type checks

d489007

Remove FrameworkSuffix header

014ca18

ozgur reviewed Feb 23, 2021

View reviewed changes

Fix indentation

b4b8bb1

elliottwilliams requested changes Feb 28, 2021

View reviewed changes

elliottwilliams self-assigned this Feb 28, 2021

Ian Stewart added 3 commits March 1, 2021 09:13

Apply framework compatibility fix from elliotwilliams

83c4a46

Tidy framework suffix handling

932eccf

Remove unnecessary guard

30dc6fd

paulb777 mentioned this pull request Mar 1, 2021

FR: Add xcframeworks in zip file on tagged assets list for version firebase/firebase-ios-sdk#7576

Closed

unarchiveAndCopyBinaryFrameworks: don't copy debug info from xcframew…

b06a5dd

…orks

tgoyne mentioned this pull request Mar 3, 2021

Add support of arm64 in Carthage build realm/realm-swift#7158

Merged

elliottwilliams approved these changes Mar 5, 2021

View reviewed changes

This was referenced Mar 6, 2021

Undefined symbols | Firestore | Carthage firebase/firebase-ios-sdk#7665

Closed

Carthage doesn't download Firebase binaries on Apple M1 firebase/firebase-ios-sdk#7039

Closed

ozgur approved these changes Mar 8, 2021

View reviewed changes

paulb777 mentioned this pull request Mar 15, 2021

Why Carthage FirebaseAnalytics only is stopped at v7.4.0 ? firebase/firebase-ios-sdk#7698

Closed

ncreated mentioned this pull request Mar 19, 2021

RUMM-1199 Fix carthage --use-xcframeworks build for DatadogCrashReporting DataDog/dd-sdk-ios#445

Merged

2 tasks

Map xcframework loading error to .readFailed()

07df005

elliottwilliams merged commit 5347dee into Carthage:master Mar 19, 2021

elliottwilliams mentioned this pull request Mar 20, 2021

Release: v0.38.0 #3152

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Xcframework prebuilt binaries #3123

Xcframework prebuilt binaries #3123

igstewart3 commented Feb 5, 2021

sebskuse commented Feb 9, 2021

dmiluski commented Feb 10, 2021 •

edited

Loading

igstewart3 commented Feb 15, 2021

finik commented Feb 23, 2021

ozgur Feb 23, 2021

igstewart3 Feb 23, 2021

ozgur Feb 23, 2021

igstewart3 Feb 23, 2021

ozgur Feb 23, 2021

igstewart3 Feb 23, 2021

elliottwilliams left a comment

elliottwilliams Feb 25, 2021

igstewart3 Mar 1, 2021

igstewart3 commented Mar 1, 2021

elliottwilliams commented Mar 3, 2021

igstewart3 commented Mar 3, 2021

elliottwilliams left a comment

Xcframework prebuilt binaries #3123

Xcframework prebuilt binaries #3123

Conversation

igstewart3 commented Feb 5, 2021

sebskuse commented Feb 9, 2021

dmiluski commented Feb 10, 2021 • edited Loading

igstewart3 commented Feb 15, 2021

finik commented Feb 23, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elliottwilliams left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

igstewart3 commented Mar 1, 2021

elliottwilliams commented Mar 3, 2021

igstewart3 commented Mar 3, 2021

elliottwilliams left a comment

Choose a reason for hiding this comment

dmiluski commented Feb 10, 2021 •

edited

Loading