Parallel building

[jeeftor]

I know there are some flags (which I'm not 100% sure which they are however) that allow Xcode to do something similar to a make -j4 parallel style build.

Does carthage support these flags and if so what are they, and if not can we get support for parallel building? The slowest part of my CI tasks is the Carthage building.

Thanks

[mdiep]

Carthage doesn't do any parallelization, but the individual projects that it compiles may have parallelized builds turned on in Xcode. (Those would be respected, as carthage calls through to xcodebuild.)

It would certainly be possible to add this to Carthage directly. But I'm not sure how much sense it would make because it would most likely lead to contention for CPU/RAM/disk. (I used to have a Mac Pro where disabling some of the CPU cores would lead to faster compilation because of RAM contention.)

If you're looking to speed up CI tasks, you're better off using binary builds or something like https://github.com/guidomb/carthage_cache.

[jeeftor]

Our issues is we exist inside a firewall and are not allowed to use github so binaries are no dice

Sent from my iPhone

On Feb 8, 2016, at 10:17 AM, Matt Diephouse notifications@github.com wrote:

Carthage doesn't do any parallelization, but the individual projects that it compiles may have parallelized builds turned on in Xcode. (Those would be respected, as carthage calls through to xcodebuild.)

It would certainly be possible to add this to Carthage directly. But I'm not sure how much sense it would make because it would most likely lead to contention for CPU/RAM/disk. (I used to have a Mac Pro where disabling some of the CPU cores would lead to faster compilation because of RAM contention.)

If you're looking to speed up CI tasks, you're better off using binary builds or something like https://github.com/guidomb/carthage_cache.

—
Reply to this email directly or view it on GitHub.

[bwhiteley]

I would like to see Carthage build multiple frameworks concurrently, when possible. It currently takes 25 minutes to build our dependencies with Debug configuration, and cores are nearly idle for much of that time.

[peterromfeldhk]

if the CI is running different jobs in parallel that have same deps i'm running to issue index.lock: File exists.

carthage bootstrap --no-use-binaries --platform iOS --derived-data $(mktemp -d) --cache-builds --log-path fastlane/build_output/carthage.log

[ThePragmaticArt]

@peterromfeldhk This has been an issue forever. I'm frankly baffled that anyone expects Carthage to scale if this functionality is not baked in. As soon as framework bundles are available via SPM, we're doing a full migration (rumor is framework bundles are currently in the works per some birdies at Apple). SPM's performance is just light years ahead of Carthage and far more scalable from a caching perspective considering the majority of the cache is assigned in the individual project's derived data. I would argue that the lack of this functionality nearly cripples Carthage for CI/CD without extra work going into binary deployments (which isn't always desirable in an enterprise environment where you own the code and are rapidly changing it).

The closest solution we have is the lovely nsoperation/carthage fork generously supported by another dev. This project has single handedly displayed how absurdly inefficient Carthage is at this point in time (3 years later...). Unfortunately, as of now, it has issues running against the new Swift headers in Xcode 11. My company has practically swapped to using this out of desperation to migrate away from the primary Carthage branch without migrating to something like Submodules or CocoPods.

Some people literally don't understand how important parallel tasks are to scaling...

[tmspzz]

@ThePragmaticArt feel free to suggest and implementation and submit a PR

[webcpu]

If anyone wants to see Carthage build multiple frameworks concurrently, when possible. Please check it out. https://github.com/unchartedworks/ConcurrentCarthage

[mikehouse]

Hi guys, I also encountered on an inability to parallelize Carthage builds. And finally I've been able to parallelize it in bash script.
Before (when PARALLEL_BUILD_FEATURE=false)

./build_carthage.sh  125.88s user 55.65s system 140% cpu 2:08.93 total
./build_carthage.sh  125.89s user 53.02s system 140% cpu 2:07.04 total
./build_carthage.sh  125.96s user 53.02s system 139% cpu 2:07.85 total

After (when PARALLEL_BUILD_FEATURE=true)

./build_carthage.sh  141.94s user 69.74s system 379% cpu 55.748 total
./build_carthage.sh  141.37s user 69.17s system 377% cpu 55.704 total
./build_carthage.sh  142.56s user 71.24s system 381% cpu 56.082 total

As you can see it is twice faster on my machine.
My Cartfile

github "psharanda/Atributika" == 4.9.10
github "CocoaLumberjack/CocoaLumberjack" == 3.5.3
github "scinfu/SwiftSoup" == 2.3.2
github "amplitude/Amplitude-iOS" == 8.17.1
github "SnapKit/SnapKit" "5.0.0"
github "SDWebImage/SDWebImage" "4.4.0"
github "devicekit/DeviceKit" == 5.0.0
binary "https://raw.githubusercontent.com/AppsFlyerSDK/AppsFlyerFramework/master/Carthage/appsflyer-ios.json" == 6.5.4

Bash script

#!/bin/sh

echo $(xcodebuild -version)

XCCONFIG=/tmp/$(uuidgen).xcconfig

echo 'BUILD_LIBRARY_FOR_DISTRIBUTION=YES' >> $XCCONFIG
echo 'ENABLE_BITCODE=NO' >> $XCCONFIG
echo 'CLANG_ENABLE_CODE_COVERAGE=NO' >> $XCCONFIG
echo 'IPHONEOS_DEPLOYMENT_TARGET=14.1' >> $XCCONFIG
echo 'MACH_O_TYPE=staticlib' >> $XCCONFIG

export XCODE_XCCONFIG_FILE="$XCCONFIG"

echo $XCCONFIG
cat $XCCONFIG

# (Re)Fetch all projects from Cartfile to ./Carthage/Checkouts with no building it.
# Cartfile.resolved will be created or updated.
carthage update --no-build

# Here you can write modification logic for fetched sources before they built.

# Fix bug for `Amplitude-iOS` https://github.com/amplitude/Amplitude-iOS/issues/411
rm ./Carthage/Checkouts/Amplitude-iOS/Amplitude.xcodeproj/xcshareddata/xcschemes/Amplitude.xcscheme || true

# You may disalbe it if it doesn't work for you.
PARALLEL_BUILD_FEATURE=true
BUILD_STATUS=${XCCONFIG}.status

function carthage_build_binary() {
    carthage build --use-xcframeworks --no-use-binaries --cache-builds --platform iOS $1
    local STATUS_CODE=$?
    if [[ $STATUS_CODE -ne 0 ]]; then
        echo $STATUS_CODE > ${BUILD_STATUS}
    fi
}

if [[ $PARALLEL_BUILD_FEATURE == true ]]; then
    while read line; do 
        if echo "${line}" | grep -q -E "^github .+"; then
            LIB_NAME=$(echo ${line} | grep -Eo "/.+\" ")
            LIB_NAME_LEN=$(($(echo $LIB_NAME | wc -c) - 1))
            LIB_NAME_FROM_INDEX=1
            LIB_NAME_TO_INDEX=$(($LIB_NAME_LEN - 2))
            LIB_NAME="${LIB_NAME:LIB_NAME_FROM_INDEX:LIB_NAME_TO_INDEX}"
            if [[ "$LIB_NAME" == "" ]]; then
                exit 1
            fi
            carthage_build_binary ${LIB_NAME} &
        elif echo "${line}" | grep -q -E "^binary .+"; then
            LIB_NAME=$(echo ${line} | grep -Eo "/[a-zA-Z0-9-]+\.json")
            LIB_NAME_LEN=$(($(echo $LIB_NAME | wc -c) - 1))
            LIB_NAME_FROM_INDEX=1
            LIB_NAME_TO_INDEX=$(($LIB_NAME_LEN - 6))
            LIB_NAME="${LIB_NAME:LIB_NAME_FROM_INDEX:LIB_NAME_TO_INDEX}"
            if [[ "$LIB_NAME" == "" ]]; then
                exit 1
            fi
            carthage_build_binary ${LIB_NAME} &
        fi
    done < "./Cartfile"
    wait
else
    # Build the projects according Cartfile.resolved
    carthage_build_binary
fi

rm $XCCONFIG

if [[ -f ${BUILD_STATUS} ]]; then
    rm ${BUILD_STATUS}
    exit 1
fi

exit 0