New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Carthage is potentially 50% slower than it needs to be #1360

Closed
schwa opened this Issue Jun 21, 2016 · 23 comments

Comments

Projects
None yet
@schwa

schwa commented Jun 21, 2016

(This is a bit of a meta-ticket - sorry)

Carthage has developed a bit of a reputation (inside my company and among other users I know who use it) for being slow. Too slow. Slow enough that it can often cause CI jobs to fail (#1310). Slow enough that someone decided a solution was to make carthage_cache to prevent Carthage from running at all on CI (https://github.com/guidomb/carthage_cache)

I wanted to know just what Carthage was doing so I took a build of Carthage and added logging to ReactiveTask so that all NSTasks would get logged to disk.

For my current day job project of 14 dependencies a Carthage build would take 1631 seconds (~27 minutes) (on a 3.1Ghz MBP with eSATA SSD).

After collecting the log file and removing everything in it that wasn't a xcodebuild clean build or dwarfdump, lipo or dsymutil. I "ran" the log and the same build took 813 seconds (~14 minutes).

So this experiment shows that Carthage might be almost twice as slow as it could be.

Furthermore by watching Activity Monitor during the build you could plainly see that the CPU on the machine were mostly at 100% during the process. Whereas watching the monitor during normal Carthage runs the CPUs spend a significant portion of their time idle (It's my understanding that a lot of this idle time is spend in timeouts waiting for xcodebuild -list/xcodebuild -showBuildSettings to return.)

Is there any work being done to reduce the total build time of Carthage? Is this something the maintainers agree is a problem and would like to address?

If much of the slow down is due to polling xcodebuild -list and/or xcodebuild -showBuildSettings could this be solved by not calling xcodebuild info so often, possibly caching that data for known versions of repositories, or perhaps by requiring the user to specify the info in the Cartfile?

@schwa schwa changed the title from Carthage is potentially 50% slower than it need to be to Carthage is potentially 50% slower than it needs to be Jun 22, 2016

@mdiep

This comment has been minimized.

Member

mdiep commented Jun 22, 2016

Only 50% slower? ☺️

Is there any work being done to reduce the total build time of Carthage? Is this something the maintainers agree is a problem and would like to address?

I absolutely would love for Carthage to be faster. I'd consider it completely unoptimized.

There are projects that cache built products (https://github.com/guidomb/carthage_cache, https://github.com/146BC/RomeBuild) and there's been discussion about caching built products (#1222). But those are primarily meant to address the slow speed of compiling—not the speed of Carthage itself.

I agree that Carthage should be faster. Some things are obviously slow. The actual resolver is very naïve, which makes it slow, for instance.

But:

  1. Correctness trumps speed
  2. There are small number of contributors
  3. No one gets paid to work on Carthage AFAIK
  4. My time is finite

So I can't make any guarantees about when Carthage will be optimized. But if you'd like for that to happen sooner, you could consider either:

  1. Opening a pull request that helps
  2. Sponsoring development so that a contributor can devote time to this specifically
@schwa

This comment has been minimized.

schwa commented Jun 22, 2016

Totally reasonable. There's no blame or accusations here.

I'm more wondering - is this something that the Carthage team (although it sounds like you're mainly the team?) is concerned about? And has any work been done to profile Carthage and find out what the hotspots are.

I'm unable to really commit any time to submitting all but the most trivial of pull requests and/or performing research & feedback (such as this bug report) but sponsorship could well be an option - If any Carthage developers are willing to receive sponsorship for improving Carthage performance that could work well for everyone. I'll see if my company can provide some budget here. Will let you know.

I think addressing this "missing" ~50% might be a good first attempt to improve Carthage speeds. Carthage should really be totally pegging my CPUs during builds and it just isn't there yet. In testing (and touched on in various tickets) I'm really wondering if we can solve the xcodebuild -list type issues. I can do more testing - I'm thinking of replacing xcodebuild -list with a tools that can cache that info. If that dramatically improves build times I'll let you know.

@mdiep

This comment has been minimized.

Member

mdiep commented Jun 22, 2016

I'm more wondering - is this something that the Carthage team (although it sounds like you're mainly the team?) is concerned about?

@ikesyo has also been a very active contributor.

This is something I'm concerned about. But I hadn't noticed such a dramatic slowdown in my usage of Carthage—probably because many of the projects I use attach pre-built binaries to GitHub Releases.

But there are a few issues I'm currently more concerned about (#1354, #1229, e.g.).

And has any work been done to profile Carthage and find out what the hotspots are.

Not that I'm aware of.

Carthage should really be totally pegging my CPUs during builds and it just isn't there yet.

Yes

I'm really wondering if we can solve the xcodebuild -list type issues. I can do more testing - I'm thinking of replacing xcodebuild -list with a tools that can cache that info. If that dramatically improves build times I'll let you know.

We cache some info in Carthage already (CachedVersions in Project.swift). Adding some more is probably a good way to start.

I also suspect that the Resolver might be doing more work in the background. There may be an easy win there. I'll try to look at that this week.

@NachoSoto

This comment has been minimized.

Contributor

NachoSoto commented Jun 25, 2016

I just noticed something that's probably contributing to this. I happened to be looking at the build logs, and realized that Carthage built ReactiveCocoa iOS at least twice 😮 I haven't dug into this at all yet, but my first idea is that it's because it's both a dependency of my project and a dependency of a dependency.

Some more details:

  • carthage version: 0.17.1
  • Command: carthage update AsyncImageView --no-use-binaries --platform ios,tvos,watchOS
  • My Cartfile:
github "ReactiveCocoa/ReactiveCocoa" ~> 4.2
github "NachoSoto/AsyncImageView" "master"
@bachirelkhoury

This comment has been minimized.

bachirelkhoury commented Jul 1, 2016

Super slow unfortunately, that we've decided to go back to how our ancestors did it, especially small libraries; manually download the library and files and add them to the project.
Project can be up and running in second anywhere, nothing to setup or download. If github is not reachable for some reason, we'll still be able to compile in the future.

The alternatives are becoming ridiculous time wasters, super slow, hangs all the time and there's always someone in the team having problem updating.

@RustyKnight

This comment has been minimized.

RustyKnight commented Jul 13, 2016

I have a setup using a customised repo for CocoaLumberjack, CocoaLumberjack, SwiftEventBus and SwiftyJSON for Swift 3 support and using realm-cocoa.git and CocoaAsyncSocket directly which is currently taking 16mins to build ... these form the core backbone of a dependent (custom) library, so each time I update the library, I'm in for another 16min wait ... not a fun solution.

I'd be nice to have some way to either skip building unchanged dependencies or cache them in some way so I could share them across projects on the machine they were built, not a fan of putting binaries into the repo, but that's me

@mdiep

This comment has been minimized.

Member

mdiep commented Jul 13, 2016

I'd be nice to have some way to either skip building unchanged dependencies or cache them in some way so I could share them across projects on the machine they were built, not a fan of putting binaries into the repo, but that's me

You can pass a list of dependencies to build or update. carthage build CocoaLumberjack

@RustyKnight

This comment has been minimized.

RustyKnight commented Jul 13, 2016

This is true, but if I need to build my library, it wants to build its dependencies, which is the problem. Carthage is awesome and I enjoy using it, but rebuilding dependencies which haven't changed is annoying.

Let me add some context: I have an internal library which needs to updated on a irregular bases, yesterday I made no less the 10 separate changes, which I need to update in the main app, so I wasted at least 2.5 hours in just re-building the libraries dependencies for the app, when they haven't changed.

I moved from CocoaPods to carthage (partly because of the simplicity), but because it allowed me to make temporary forks of the third party libraries to incorporate Swift 3 changes with Xcode 8 (I have no choice, I've been forced onto this train). I like carthage, for the most part, it doesn't get in the way and it works, but this is driving me nuts. I've made release archives for my "personalized" forks of this third party libraries to maintain my sanity, but given the issues with Xcode and Swift frameworks, I'd like to avoid it into the future, at least until Apple actually makes it possible to deliver Swift binary frameworks :P

I'd imagine a scenario which meant that once the frameworks were built for a given project, they wouldn't need to be rebuilt unless a change was detected (let's keep it simple, a version change in the repo's tag or change in the head); I'd love to see a scenario where they didn't need to be rebuilt for the local machine, but I'll live with the first

@hardikdevios

This comment has been minimized.

hardikdevios commented Jul 15, 2016

Ok so i am not the only one who thinks that Carthage is slow. Here are few things that you guys might already facing.
#1291
#1288
So i ended up using carthage and decided to go back to my older approach which at least give me more flexibility and reduce my build time. i guess Carthage is not yet optimized if you have larger number of dependencies. It is very tough to mange(add,update,delete) but i guess something good will going to come for sure in near future for Carthage.

@dmcrodrigues

This comment has been minimized.

dmcrodrigues commented Aug 1, 2016

@mdiep I'm also seeing the same issue referred by @NachoSoto while updating Rex, which only depends of ReactiveCocoa.

In the log file we can check that ReactiveCocoa was built 4 times, 2 per SDK, which seems to explain why it takes so much time to update. There's a plausible reason which can explain this?

*** Building scheme "ReactiveCocoa-iOS" in ReactiveCocoa.xcworkspace
/usr/bin/xcrun xcodebuild -workspace Rex/Carthage/Checkouts/ReactiveCocoa/ReactiveCocoa.xcworkspace -scheme ReactiveCocoa-iOS -configuration Release -sdk iphoneos ONLY_ACTIVE_ARCH=NO BITCODE_GENERATION_MODE=bitcode CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES clean build
…
** BUILD SUCCEEDED **

/usr/bin/xcrun xcodebuild -workspace Rex/Carthage/Checkouts/ReactiveCocoa/ReactiveCocoa.xcworkspace -scheme ReactiveCocoa-iOS -configuration Release -sdk iphoneos ONLY_ACTIVE_ARCH=NO BITCODE_GENERATION_MODE=bitcode CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES clean build
…
** BUILD SUCCEEDED **

/usr/bin/xcrun xcodebuild -workspace Rex/Carthage/Checkouts/ReactiveCocoa/ReactiveCocoa.xcworkspace -scheme ReactiveCocoa-iOS -configuration Release -sdk iphonesimulator -destination platform=iOS Simulator,id=DAAB8B95-3328-4215-BA9D-ECF30E21762C -destination-timeout 3 ONLY_ACTIVE_ARCH=NO BITCODE_GENERATION_MODE=bitcode CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES clean build
…
** BUILD SUCCEEDED **

/usr/bin/xcrun xcodebuild -workspace Rex/Carthage/Checkouts/ReactiveCocoa/ReactiveCocoa.xcworkspace -scheme ReactiveCocoa-iOS -configuration Release -sdk iphonesimulator -destination platform=iOS Simulator,id=DAAB8B95-3328-4215-BA9D-ECF30E21762C -destination-timeout 3 ONLY_ACTIVE_ARCH=NO BITCODE_GENERATION_MODE=bitcode CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES clean build
…
** BUILD SUCCEEDED **
@mdiep

This comment has been minimized.

Member

mdiep commented Aug 1, 2016

That's because Rex builds its own copy of ReactiveCocoa in its workspace instead of using the one that Carthage built.

@dmcrodrigues

This comment has been minimized.

dmcrodrigues commented Aug 1, 2016

@mdiep I'm not following, how Rex is relevant in a carthage update when Carthage is self-contained?

@mdiep

This comment has been minimized.

Member

mdiep commented Aug 1, 2016

  1. Carthage builds RAC directly because it's a dependency of Rex
  2. Carthage builds Rex through its workspace
    1. Xcode builds RAC because it's in the Rex workspace as a dependency
    2. Xcode builds Rex in the workspace
@dmcrodrigues

This comment has been minimized.

dmcrodrigues commented Aug 1, 2016

@mdiep I'm not referring to building Rex, I'm only focused in your step 1) and in that step, while running carthage update --platform iOS --no-use-binaries, I'm seeing Carthage building RAC 4 times, 2 times using the iPhoneOS SDK and 2 times using the iPhoneSimulator SDK.

I understand why we need to build using both SDKs, what I do not understand is why is being built more than once per SDK and if that is expected or not, because it may be one of the reasons behind this issue (bad performance).

@dmcrodrigues

This comment has been minimized.

dmcrodrigues commented Aug 1, 2016

I've done a few more tests using other projects and seems this issue of building the same framework more than once is something within ReactiveCocoa project but I still don't know the exact reason behind it.

@RustyKnight

This comment has been minimized.

RustyKnight commented Aug 1, 2016

When building Realm, I noticed it was build the workspace, not the project, which also built the examples, which really aren't needed. It'd be nice to also control what Carthage used (specify the project/workspace)

@schwa

This comment has been minimized.

schwa commented Aug 3, 2016

I've added the ability to skip schemes & projects to Punic - my workalike replacement for Carthage: see https://github.com/schwa/punic#punicyaml-skip-lists - let me know if that works for you.

In testing with our work project skipping unneeded schemes speeds up builds by about 40%. On top of that punic won't unnecessarily clean the projects unless you tell it - which can dramatically reduce build times.

I'd love to see these features make it in Carthage too.

@tcurdt

This comment has been minimized.

tcurdt commented Aug 19, 2016

You can pass a list of dependencies to build or update. carthage build CocoaLumberjack

@mdiep if only that would work #1418

@groue

This comment has been minimized.

groue commented Sep 20, 2016

The slowness of Carthage impacts other repositories, because Carthage users ask library developers for pre-built binaries (1, 2, ...). It is not really cool to have to ship pre-built binaries, and it is not really cool to answer "this is a Carthage issue" either.

The main problem looks like carthage update rebuilds everything, and people don't know how to avoid that (see related stackOverflow question).

@ikesyo

This comment has been minimized.

Member

ikesyo commented Sep 23, 2016

#1360 (comment): This works as expected.

@tcurdt

This comment has been minimized.

tcurdt commented Sep 23, 2016

@ikesyo sorry - but that's only true for build and not for update.

@calebd

This comment has been minimized.

calebd commented Sep 29, 2016

This is still a problem.

A Cartfile with only the contents github "ReactiveCocoa/ReactiveCocoa" results in the following output in the build log:

=== BUILD TARGET Result-iOS OF PROJECT Result WITH CONFIGURATION Release ===
=== BUILD TARGET Result-iOS OF PROJECT Result WITH CONFIGURATION Release ===
=== BUILD TARGET Result-iOS OF PROJECT Result WITH CONFIGURATION Release ===
=== BUILD TARGET ReactiveCocoa-iOS OF PROJECT ReactiveCocoa WITH CONFIGURATION Release ===
=== BUILD TARGET Result-iOS OF PROJECT Result WITH CONFIGURATION Release ===
=== BUILD TARGET ReactiveCocoa-iOS OF PROJECT ReactiveCocoa WITH CONFIGURATION Release ===
=== BUILD TARGET Result-iOS OF PROJECT Result WITH CONFIGURATION Release ===
=== BUILD TARGET ReactiveCocoa-iOS OF PROJECT ReactiveCocoa WITH CONFIGURATION Release ===
=== BUILD TARGET Result-iOS OF PROJECT Result WITH CONFIGURATION Release ===
=== BUILD TARGET ReactiveCocoa-iOS OF PROJECT ReactiveCocoa WITH CONFIGURATION Release ===
@mdiep

This comment has been minimized.

Member

mdiep commented Sep 30, 2016

I was able to reproduce the duplicate build problem.

Carthage loads the builds settings for the scheme in the ReactiveCocoa workspace. It's finding settings for both Result.framework and ReactiveCocoa.framework since they're both explicitly listed in the scheme, which results in the double build.

I'm looking into how to fix this particular issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment