Framework binary file is too large

[ceyhun]

Hello,
I was using TwilioVideo 1.0.0-beta1 via CocoaPods and it had a binary size of ~56 MB. Now I updated to 1.0.0-beta14 and it became ~155 MB.

I commit my /Pods directory to source control and this is not acceptable to commit and push each time I update the framework.

Can it be because you are using Xcode 8.3 to build it. Because there is a radar about this issue http://www.openradar.me/31302382.

If that's the case you should switch to Xcode 8.3.1 which fixed this problem https://developer.apple.com/library/content/releasenotes/DeveloperTools/RN-Xcode/Chapters/Introduction.html
They will soon be deprecating Xcode 8.3, at which time app archives built with Xcode 8.3 will no longer be accepted by the App Store.

[theprojectabot]

git lfs might be a solution for you

[ceyhun]

I don't want git lfs as a dependency for my project. I just want to be able to git clone and run it.
I don't want every single developer on the team to install git lfs just for this.
I don't want my app to grow in size because of this huge framework.
There is clearly a problem with the binary being this large. I think it can be fixed and it should be fixed.

[ceaglest]

Hi @ceyhuno,

The reason why the binary size increased significantly in 1.0.0-beta13 is because we added support for Bitcode. Unfortunately, there is not much that we can do to decrease the file size of our SDK given Apple's preference that App Store submissions should include Bitcode. This is not a hard requirement yet, but knowing Apple there will be a time soon when Bitcode support becomes mandatory. For reference we do not use Xcode 8.3.0 on our build servers.

#71

Have you considered not checking in your /Pods directory and relying on the Podfile.lock to fetch the right dependencies?

Best,
Chris Eagleston

[langford]

remote: error: File Pods/TwilioVideo/TwilioVideo.framework/TwilioVideo is 148.54 MB; this exceeds GitHub's file size limit of 100.00 MB

Cocoapods recommends checking in pods. I understand this is different than many other languages do with their dependency managers.

Your framework exceeds github's allowable size

Among other reasons, people check in pods because companies, especially companies with closed source frameworks, can arbitrarily remove them at any time, sometimes accidentally.

This has happened several times in the iOS community, breaking hundreds of people's apps builds and making people miss many commitments to their customers.

Checking in pods also allows for a reliable build when the network is down, you are on a plane, and for quick CI builds.

I suggest breaking the framework up into multiple frameworks.

[ceaglest]

Hi @langford,

We are certainly not done with optimizing the file size of our SDK, however I can't promise that we will be able to reduce it below 100 MB (including Bitcode). I will have a chat with the iOS team members to discuss your suggestion of splitting the framework into multiple pieces.

Among other reasons, people check in pods because companies, especially companies with closed source frameworks, can arbitrarily remove them at any time, sometimes accidentally.

This has happened several times in the iOS community, breaking hundreds of people's apps builds and making people miss many commitments to their customers.

Checking in pods also allows for a reliable build when the network is down, you are on a plane, and for quick CI builds.

I would recommend that you fork open source projects that you consume via Cocoapods. As for closed source dependencies you can always mirror them if you are worried that they might go away. Ultimately, in order to get your working copy up and running the bits which represent your dependencies need to be fetched from somewhere. Wether that involves git or a combination of git and pod install is up to you.

As for Twilio, we are not in the habit of removing old releases from our CDN or other distribution channels like Cocoapods. The only case where we might do this is for a deprecated Client SDK which can no longer talk to our production infrastructure. If you ever have any issues downloading a release of our SDKs please do let us know.

Strip Bitcode via Podfile

In the short term, the best solution I can offer for you and @ceyhuno is to strip the Bitcode symbols as part of your Cocoapods installation process. This will allow a similar binary size to releases prior to 1.0.0-beta13, but without Bitcode support for app store submissions. Here is an example of the Objective-C Quickstart Podfile modified to perform this. Please excuse my lack of Ruby polish.

source 'https://github.com/twilio/cocoapod-specs'

target 'ObjCVideoQuickstart' do
  pod 'TwilioVideo', '1.0.0-beta14'
end

post_install do |installer|
  # Find bitcode_strip
  bitcode_strip_path = `xcrun -sdk iphoneos --find bitcode_strip`.chop!

  # Find path to TwilioVideo dependency
  path = Dir.pwd
  framework_path = "#{path}/Pods/TwilioVideo/TwilioVideo.framework/TwilioVideo"

  # Strip Bitcode sections from the framework
  strip_command = "#{bitcode_strip_path} #{framework_path} -m -o #{framework_path}" 
  puts "About to strip: #{strip_command}"
  system(strip_command)
end

Best,
Chris Eagleston

[ceyhun]

Thank you for the Bitcode strip workaround. I hope soon you can reduce it below 100 MB (including Bitcode) or find a way to split it into multiple pieces at least before Bitcode support becomes mandatory.

[langford]

@ceaglest This is 100% on point technical customer service. This immediately solved my problem for the short term. The above with a rebase -i and now everything is happy on GitHub yet again

Thank you for your help.

The remainder of this comment is customer experience sharing, to help with any internal conversations you have as to this matter.

As for Twilio, we are not in the habit of removing old releases from our CDN or other distribution channels like Cocoapods

I imagine you're far less likely to do it than many other companies (who've only sometimes done it related to a CRM change, or something like that). You're a longstanding company that's served developers well. Once we move to the "don't check pods in" method...we have to rely on all the vendors (and retool CI builds and expectations around build times).

You can provide a high level of library archival, but most companies cannot be relied on to support that, if they even still exist a couple years down the line. Many -- if not the vast majority of iOS apps -- do not generate enough revenue to justify full time engineers on them. Even 1-2 year gaps on development is very common.

Your mitigation approaches are ones I've tried in several organization. They're resource intensive (engineers cost more than repos) compared to checking in the pods, but do work. Swapping to the no-check-in approach means the other pods we rely on require backup and "librarian procedures" too.

Even removing files years later though can break someone's build so that when they hire a consultant to rev it, it can't even be compiled even to check to see if it's all there. A building app that fails to work properly is a very different problem than one that can't even build or work properly.

This means sometimes, people can't hire a consultant even, as the consultant wants to see the code. It doesn't build, so consultant passes on the contract as they can't tell if there is budget to fix it or if "it" is all there.

[whench18]

Hello, you can try using the long path tool. This may help you with this too big file you have.

[ceyhun]

@whench18 what exactly is long path tool? What does it do? Can you give a link for that?

[seabjackson]

Hi is there any update to getting this to work. I have tried lfs and can't seem to configure. The tentative solution above doesn't seem to work anymore. I am using Xcode 10 on Mojave. Any insight into how to push to github with file exceeding 100MB would be greatly appreciated.

[piyushtank]

@seabjackson

@ceaglest and @paynerc are on vacation and he will be back early next month. We will look into the issue and get back when team is back from vacation.

[ceaglest]

Hi @seabjackson,

Hi is there any update to getting this to work. I have tried lfs and can't seem to configure. The tentative solution above doesn't seem to work anymore.

We don't officially document/support stripping of Bitcode segments because of Apple's App Store submission requirements. That being said, I'd be happy to take another look at the Podfile I posted above if it will help in this case.

Best,
Chris

[ceaglest]

Hi @seabjackson,

I had another look and was able to get the bitcode stripping to work with some small modifications. In more recent versions of Twilio Video, the structure of our artifact has changed in order to support distribution via Carthage.

Full code - https://github.com/ceaglest/video-quickstart-swift/tree/issue/110-framework-size

Stripping is done as a post install step, like so:

post_install do |installer|
  # Find bitcode_strip
  bitcode_strip_path = `xcrun -sdk iphoneos --find bitcode_strip`.chop!

  # Find path to TwilioVideo dependency
  path = Dir.pwd
  framework_path = "#{path}/Pods/TwilioVideo/Build/iOS/TwilioVideo.framework/TwilioVideo"

  # Strip Bitcode sections from the framework
  strip_command = "#{bitcode_strip_path} #{framework_path} -m -o #{framework_path}" 
  puts "About to strip: #{strip_command}"
  system(strip_command)
end

My apologies that there is no error handling, but this does indeed strip the bitcode segments and reduces the size of 2.6.0 to 60,383,727 bytes on my disk.

Regards,
Chris

[jagged91]

@ceaglest Is the same bitcode stripping possible via Carthage? I'm running into the same issue (SDK too large to commit to Github).

[ceaglest]

Hi @jgadsby,

I don't think Carthage supports post install hooks like CocoaPods does. Instead, you could modify your workspace setup script (or manual steps) like this:

1. carthage update
2. Strip bitcode using bitcode_strip similar to the script posted above.
3. Continue to your next steps ....

Best,
Chris

[ceaglest]

Hi Developers,

If part of your workflow involves committing the Twilio Video binary, and the size is a problem then you might be interested in trying our 3.x releases. We've dropped support for 32-bit architectures, which reduces the size of the binary by roughly 50%.

3.0.1 is available now.

Best,
Chris

[ceaglest]

Twilio Video 3.x releases are smaller than 100 MB un-compressed, and bitcode has been a requirement for app store submission for a long time. If you want to commit the framework to source control we recommend migrating to from 2.x to 3.x.