Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Update] Cloud Firestore Xcode Build Times #2751

Closed
Salakar opened this issue Jun 12, 2020 · 103 comments
Closed

[Update] Cloud Firestore Xcode Build Times #2751

Salakar opened this issue Jun 12, 2020 · 103 comments

Comments

@Salakar
Copy link
Member

Salakar commented Jun 12, 2020

Hey everyone 馃憢

One of the key issues we wanted to look into providing a solution for was #349 - improving build times for Xcode when using Firestore.

I'm happy to say we now have something for you to try (please provide feedback) that should significantly improve your build times;

Mac mini (2018) 6 cores:

Before:    ~ 240s
After:     ~  45s

GitHub Action CI 2 cores:

Before:    ~ 551s
After:     ~ 174s

Why is it slow?

Currently the Firestore iOS SDK depends on some 500k lines of mostly C++ code (from dependencies such as gRPC), which when compiling as part of your Xcode build can take a long time - even more so in CI environments.

How can we speed it up?

The biggest win to improving build times would be to not have to compile all that C++ code. The way to do that would be to consume precompiled binaries, in the new XCFramework bundle type introduced to Xcode last year, luckily as part of the Firebase iOS SDK release process these bundles are generated and published in a zip containing all Firebase SDKs.

To help you consume these quicker with the least amount of changes to your codebase we've created invertase/firestore-ios-sdk-frameworks - a repository that provides a drop in replacement podspec for Firestore that consumes the precompiled binaries instead, without you needing to download 3gb of SDKs and manually extract out the Firestore SDKs from the rest of the SDKs in the zip.

Setup

To integrate, you just need to add 1 line to your ios/Podfile inside your Flutter project;

pod 'FirebaseFirestore', :git => 'https://github.com/invertase/firestore-ios-sdk-frameworks.git', :tag => '7.4.0'

Add this line inside your target 'Runner' do block in your Podfile, e.g.:

# ...
target 'Runner' do
  pod 'FirebaseFirestore', :git => 'https://github.com/invertase/firestore-ios-sdk-frameworks.git', :tag => '7.4.0'

# ...
end

No changes to the FlutterFire packages were required. See #4698 for a full example of integrating with automatic versioning based on what firebase_core uses.

Why not ship the binaries with the pub packages?

This would add ~200mb to every release. Whereas via a custom pod this is only ever downloaded once on your machine per Firebase iOS SDK version.


For more information on the pod and versioning please see the invertase/firestore-ios-sdk-frameworks repository.


鈿狅笍 Note: if you were previously caching iOS builds on CI you may now find that when using precompiled binaries that caching is no longer required and it may actually slow down your build times by several minutes. See this comment below for more; #2751 (comment)


Please let me know if this has helped you, and I would love to see some before and after timings if you feel like sharing

@Salakar Salakar changed the title Cloud Firestore Xcode Build Times [Update] Cloud Firestore Xcode Build Times Jun 12, 2020
@Salakar Salakar pinned this issue Jun 12, 2020
@Salakar Salakar self-assigned this Jun 12, 2020
@mikehardy
Copy link
Contributor

mikehardy commented Jun 12, 2020

This actually works for more than flutter, for react-native-firebase or anyone that integrates the firestore pod it works, I know this because my creaky old 4 core mbp takes about 5mins less to compile and get the app running. 1 line! Thank you.

@Salakar
Copy link
Member Author

Salakar commented Jun 12, 2020

This actually works for more than flutter, for react-native-firebase or anyone that integrates the firestore pod it works, I know this because my creaky old 4 core mbp takes about 5mins less to compile and get the app running. 1 line! Thank you.

Yep, works for anyone using Firestore on iOS - no matter the framework 馃帀

@galvin59
Copy link

galvin59 commented Jun 12, 2020

Thanks for the tip.
However when I try to implement it, I get :

 [!] CocoaPods could not find compatible versions for pod "FirebaseFirestore":
      In snapshot (Podfile.lock):
        FirebaseFirestore (= 1.11.2, ~> 1.11.2)
      In Podfile:
        FirebaseFirestore (from `https://github.com/invertase/firestore-ios-sdk-frameworks.git`, tag `6.26.0`)
    None of your spec sources contain a spec satisfying the dependencies: `FirebaseFirestore (from `https://github.com/invertase/firestore-ios-sdk-frameworks.git`, tag `6.26.0`), FirebaseFirestore (= 1.11.2, ~> 1.11.2)`.
    You have either:
     * out-of-date source repos which you can update with `pod repo update` or with `pod install --repo-update`.
     * mistyped the name or version.
     * not added the source repo that hosts the Podspec to your Podfile.

I tried to pod update, but with no luck

Any idea ?

@Ehesp
Copy link
Member

Ehesp commented Jun 12, 2020

@jsoms try removing your Podfile.lock and trying again.

@orestesgaolin
Copy link

orestesgaolin commented Jun 12, 2020

I started simple tests on codemagic and there's at least few minutes decrease in iOS build time. Unfortunately cache upload caused the total build to be still rather long.

I'm going to update this table with next results.

Flutter stable 1.17.3
Xcode 11.5

Id Precompiled binaries Cache machine cloud_firestore version Firestore Fetch Installing dep iOS pod install Xcode build Total
40 馃毇 (no_cache) no mini 0.13.4+2 6.23.0 0:00:04 0:01:43 0:30:50 278 793 0:49:34
41 馃毇 (no_cache) no mini 0.13.4+2 6.23.0 0:00:04 0:01:09 0:26:10 212 755 0:45:10
251 馃毇 (cache) yes mini 0.13.4+2 6.23.0 0:03:10 0:01:01 0:27:43 291 604 0:48:20
253 (cache) yes mini 0.13.4+2 6.23.0 0:04:28 0:01:27 0:16:17 193 402 0:42:00
254 (cache) yes mini 0.13.4+2 6.23.0 0:02:26 0:00:56 0:15:00 195 376 0:36:30
255 (cache) yes mini 0.13.4+2 6.23.0 0:04:01 0:01:17 0:14:55 201 424 0:41:33
1_no_cache (no_cache) no mini 0.13.4+2 6.23.0 0:00:05 0:01:09 0:13:29 165 404 0:25:52
2_no_cache (no_cache) no mini 0.13.4+2 6.23.0 0:00:18 0:01:27 0:14:43 169 478 0:26:06
3_no_cache (no_cache) no mini 0.13.4+2 6.23.0 00:00:07 00:02:15 00:17:04 220 463 00:30:20
4_no_cache_pro (no_cache_pro) no pro 0.13.4+2 6.23.0 00:00:02 00:00:57 00:09:27 141 273 00:17:21
5_no_cache (no_cache) no mini 0.13.4+2 6.23.0 00:00:05 00:01:11 00:12:39 136 366 00:23:52

image

1_no_cache is completely new workflow on codemagic without any cache but with precompiled binaries!

pro means that it was run on Mac Pro

Important remark when using cache the Android build is significantly shorter (eg 15 vs 9 min). Also cache usually takes 2 min to download and 4-6 min to create and upload.

@Salakar
Copy link
Member Author

Salakar commented Jun 12, 2020

@orestesgaolin what's it like without uploading build caches?

@orestesgaolin
Copy link

orestesgaolin commented Jun 12, 2020

@Salakar I updated my comment above and it's so surprising to see that without any cache (i.e. completely clean build on new workflow) the build time dropped dramatically to about 13 min. I'm gonna run it again without any cache

@orestesgaolin
Copy link

orestesgaolin commented Jun 12, 2020

@Salakar second build without cache was a bit longer, so it seems that it doesn't have significant effect (thus it may be completely unnecessary to use). I will be collecting data in subsequent builds and maybe we can then draw some conclusions out of it

@jmagman
Copy link
Contributor

jmagman commented Jun 12, 2020

Note using xcframeworks requires CocoaPods 1.9.0 or higher. Flutter currently requires a minimum of 1.6.

@Salakar Created issue to add cocoapods_version to the podspec. invertase/firestore-ios-sdk-frameworks#1.

@ElaineSchwner
Copy link

ElaineSchwner commented Jun 12, 2020

Could you help me?

  • I run 'pod deintegrate';
  • Deleted Pofile.lock
  • Run 'pod install'

And I got this:
[!] CocoaPods could not find compatible versions for pod "FirebaseFirestore":
In Podfile:
FirebaseFirestore (from https://github.com/invertase/firestore-ios-sdk-frameworks.git, tag 6.26.0)

cloud_firestore (from `.symlinks/plugins/cloud_firestore/ios`) was resolved to 0.0.1, which depends on
  Firebase/Firestore (~> 6.0) was resolved to 6.0.0, which depends on
    FirebaseFirestore (~> 1.3.0)

What must I do?
Thank you.

@red42
Copy link
Contributor

red42 commented Jun 12, 2020

@ElaineSchwner I solved a similar problem with pod repo update as per @jsoms ' comment.

@ElaineSchwner
Copy link

ElaineSchwner commented Jun 12, 2020

pod repo update

Solved !
Thank you!

@mafreud
Copy link
Contributor

mafreud commented Jun 14, 2020

It worked well. Thanks a lot 馃洨锔 馃槃

@RaviKavaiya
Copy link

RaviKavaiya commented Jun 16, 2020

This didn't work for me. I tried pod install, pod update...
See this:

Xcode build done.
Could not build the application for the simulator.
Error launching application on iPhone 11.

@Ehesp
Copy link
Member

Ehesp commented Jun 16, 2020

Hi @RaviKavaiya we'll need more information to help on that one. It looks like the XCode build succeeded but then fails when the app is launched - what are the errors?

@RaviKavaiya
Copy link

RaviKavaiya commented Jun 16, 2020

@Ehesp I tried running with flutter run -v. Couldn't find anything. Will give another shot later and will tell you..

Thanks

@andre-arsenault
Copy link

andre-arsenault commented Jun 17, 2020

Build timings on my MacBook Pro (2018) 4 cores:

Before:   ~ 272s
After:    ~ 122s

A very nice speed improvement, thank you!

How can we tell which version of the iOS SDK the cloud_firestore package uses, so we can keep the tag up to date in our Podfile? I hope that will be part of the package's release notes going forward?

@ghost
Copy link

ghost commented Jun 17, 2020

ld: library not found for -lBoringSSL-GRPC

I've raised an issue against the firestore-ios-sdk-frameworks library - referencing it here in case it helps someone else...

@farukozsoy
Copy link

farukozsoy commented Jun 19, 2020

On an old Macbook Pro ( 2011 - 2 cores - i7 )

before ~ 15 minutes
after ~ 9 minutes

Thanks!

@nilsreichardt
Copy link
Contributor

nilsreichardt commented Jun 23, 2020

Is there also a workaround for Flutter macOS?

@Salakar
Copy link
Member Author

Salakar commented Jun 23, 2020

@AndroidNils unfortunately this is only for iOS

@zumrywahid
Copy link

zumrywahid commented Mar 29, 2021

Use folliwing line for

pod 'FirebaseFirestore', :git => 'https://github.com/invertase/firestore-ios-sdk-frameworks.git', :tag => '7.3.0'

for your puspec cloud_firestore: "^0.16.0+1" version .

@mikehardy
Copy link
Contributor

mikehardy commented Mar 29, 2021

@zumrywahid a more general recommendation (as your recommendation will go out of date as versions are released) is to consult the exact location that the firebase-ios-sdk is defined for the firebase_core package

That's right here: https://github.com/FirebaseExtended/flutterfire/blob/master/packages/firebase_core/firebase_core/ios/firebase_sdk_version.rb#L2

And it's possible to inspect tags / versions on that file in case you are not using the latest version

It is currently 7.3.0 as you mention though, yes :-)

@Python3-8
Copy link

Python3-8 commented May 30, 2021

I get this error:

[!] CocoaPods could not find compatible versions for pod "FirebaseFirestore":
  In Podfile:
    FirebaseFirestore (from `https://github.com/invertase/firestore-ios-sdk-frameworks.git`, tag `8.0.0`)

    cloud_firestore (from `.symlinks/plugins/cloud_firestore/ios`) was resolved to 2.0.0, which depends on
      Firebase/Firestore (= 7.11.0) was resolved to 7.11.0, which depends on
        FirebaseFirestore (~> 7.11.0)

NOTE: I changed the tag from 7.4.0 to 8.0.0, because 7.4.0 didn't work either.

My ios/Podfile:

target 'Runner' do
  pod 'FirebaseFirestore', :git => 'https://github.com/invertase/firestore-ios-sdk-frameworks.git', :tag => '8.0.0'
  # :tag => '7.4.0'

  use_frameworks!
  use_modular_headers!

  flutter_install_all_ios_pods File.dirname(File.realpath(__FILE__))
end

My pubspec.yaml:

dependencies:
  flutter:
    sdk: flutter
  cloud_firestore: ^2.0.0
  cupertino_icons: ^1.0.2
  email_validator: ^2.0.1
  firebase_auth: ^1.1.3
  firebase_core: ^1.1.0
  flutter_icons: ^1.1.0
  shared_preferences: ^2.0.5
  intl: ^0.17.0
  scrollable_positioned_list: ^0.1.10
  provider: ^5.0.0
  flutter_markdown: ^0.6.2
  flutter_slidable: ^0.6.0

This might help as well:
I got this error while running pod update, and here's the log:

Update all pods
Updating local specs repositories
Analyzing dependencies
Pre-downloading: `FirebaseFirestore` from `https://github.com/invertase/firestore-ios-sdk-frameworks.git`, tag `8.0.0`
FirebaseFirestore/AutodetectLeveldb: subspec would include leveldb if used directly or by default.
cloud_firestore: Using Firebase SDK version '7.11.0' defined in 'firebase_core'
firebase_auth: Using Firebase SDK version '7.11.0' defined in 'firebase_core'
firebase_core: Using Firebase SDK version '7.11.0' defined in 'firebase_core'
[!] CocoaPods could not find compatible versions for pod "FirebaseFirestore":
  In Podfile:
    FirebaseFirestore (from `https://github.com/invertase/firestore-ios-sdk-frameworks.git`, tag `8.0.0`)

    cloud_firestore (from `.symlinks/plugins/cloud_firestore/ios`) was resolved to 2.0.0, which depends on
      Firebase/Firestore (= 7.11.0) was resolved to 7.11.0, which depends on
        FirebaseFirestore (~> 7.11.0)

[!] Automatically assigning platform `iOS` with version `10.0` on target `Runner` because no platform was specified. Please specify a platform for this target in your Podfile. See `https://guides.cocoapods.org/syntax/podfile.html#platform`.

Any help would be appreciated :)

@bounty1342
Copy link

bounty1342 commented May 30, 2021

You should take the version matching the one of your package.
It screams 7.11 from the logs...

@Python3-8
Copy link

Python3-8 commented May 31, 2021

Sorry, I have absolutely no experience in iOS development. What file am I supposed to edit?

@Python3-8
Copy link

Python3-8 commented May 31, 2021

I changed this:

pod 'FirebaseFirestore', :git => 'https://github.com/invertase/firestore-ios-sdk-frameworks.git', :tag => '8.0.0'

...to this:

pod 'FirebaseFirestore', :git => 'https://github.com/invertase/firestore-ios-sdk-frameworks.git', :tag => '7.11.0'

...and ran pod update. It ran successfully and I'm gonna try building my app now.

@Python3-8
Copy link

Python3-8 commented May 31, 2021

The build still takes a long time... It took 8 minutes. Is this because it was the first build since I modified the Podfile, or do I have any other dependencies that might be causing a problem?

These are my dependencies:

dependencies:
  flutter:
    sdk: flutter
  cloud_firestore: ^2.0.0
  cupertino_icons: ^1.0.2
  email_validator: ^2.0.1
  firebase_auth: ^1.1.3
  firebase_core: ^1.1.0
  flutter_icons: ^1.1.0
  shared_preferences: ^2.0.5
  intl: ^0.17.0
  scrollable_positioned_list: ^0.1.10
  provider: ^5.0.0
  flutter_markdown: ^0.6.2
  flutter_slidable: ^0.6.0

@Python3-8
Copy link

Python3-8 commented Jun 1, 2021

I think it's actually working! I just built my app again to work on it a little bit, and the build finished in 82 seconds.

Launching lib/main.dart on iPhone 12 in debug mode...
Xcode build done.                                           81.8s
Connecting to VM Service at ws://127.0.0.1:58642/...
...

@Valentin-Seehausen
Copy link

Valentin-Seehausen commented Jun 3, 2021

You should take the version matching the one of your package.
It screams 7.11 from the logs...

Could this appear in the explanatory text? I stumbled upon it and I guess an explanation would be helpful to other people, too.

@mikehardy
Copy link
Contributor

mikehardy commented Jun 3, 2021

@Valentin-Seehausen hi there! If it is missing in some location that seems appropriate to you, I think in github you can just hit the little edit button on the README.md and suggest where it should be. We'd be happy to merge any reasonable change that would help users, most of our documentation is improved in that way since we (that is, the maintainers) are usually too closed to / used to things to have the perspective of fresh users anymore.

@kasvith
Copy link

kasvith commented Jul 6, 2021

Thank you, this worked well :)

Make sure to check the actual Firebase version in Podfile

@lemonbitsapp
Copy link

lemonbitsapp commented Jul 15, 2021

guys, im using SwiftPackage manager, - the build times is usually 40-50mins thru CI/CD. this is O M G. is there a setting to speed this up. Tks.

@mikehardy
Copy link
Contributor

mikehardy commented Jul 16, 2021

Depending on your CI/CD there are other tricks to speed it up since this doesn't do SPM yet
https://github.com/hendrikmuhs/ccache-action/ or
https://github.com/mikehardy/buildcache-action

Those are for github but the general solution design ("Install compiler cache tool, make symbolic links to the tool (or update your build definition to use it directly), save and restore compiler cache across builds") is generic and I have seen it work on bitrise as well so I know it's generally applicable.

@nathanaelneveux
Copy link

nathanaelneveux commented Jul 20, 2021

Trying to implement the automatic versioning based on what firebase_core uses (per #4698) but kinda hitting a wall.

This line of course doesn't point to anything on my system - I also can't make assumptions about where it is on other members of my teams system's:
https://github.com/FirebaseExtended/flutterfire/blob/7d806d710fa6c3968373245e0b7f6bcf9dde0d96/packages/cloud_firestore/cloud_firestore/example/ios/Podfile#L41

Trying to find a replacement for this line I've come up with this:

def firebase_core
  generated_flutter_plugins_path = File.expand_path(File.join('..', '..', '.flutter-plugins'), __FILE__)
  unless File.exist?(generated_flutter_plugins_path)
    raise "#{generated_flutter_plugins_path} must exist. If you're running pod install manually, make sure flutter pub get is executed first"
  end

  File.foreach(generated_flutter_plugins_path) do |line|
    matches = line.match(/firebase_core\=(.*)/)
    return matches[1].strip if matches
  end
  raise "firebase_core not found in #{generated_flutter_plugins_path}. Try deleting .flutter-plugins, then run flutter pub get"
end

which would allow me to reference the script file like this:

firebase_core_script = File.expand_path(File.join('ios', 'firebase_sdk_version.rb'), firebase_core)

However flutter responds like this when I ty to build:

Exception: Warning: Podfile is out of date
  This can cause issues if your application depends on plugins that do not support iOS or macOS.
  See https://flutter.dev/docs/development/packages-and-plugins/developing-packages#plugin-platforms for details.
  If you have local Podfile edits you would like to keep, see https://github.com/flutter/flutter/issues/45197 for instructions.

It seems that having any mention of .flutter-plugins in your podfile causes this message to appear and not build. So I'm at a bit of a loss as to where to find a consistent reference to the firebase_core folder.

EDIT
ok - something like changing it to '.flutter'+'-plugins' gets around the flutter test.
https://github.com/flutter/flutter/blob/e028d0f046c20c61b7c7c8a23bc456f337410658/packages/flutter_tools/lib/src/macos/cocoapods.dart#L408
But the question still stands on is there a better place I should be referencing to get the location of the firebase_core folder?

@devbrandonlong
Copy link

devbrandonlong commented Aug 2, 2021

Went from

Screen Shot 2021-08-01 at 8 54 00 PM

to

Screen Shot 2021-08-01 at 8 54 13 PM

Nicely done 馃檶馃徑

@misteroak
Copy link

misteroak commented Aug 14, 2021

Yes, significant improvement. Went from 568s to 32s (MBP 3.1 GHz Quad-Core Intel Core i7). Thank you!

@dreadjr
Copy link

dreadjr commented Aug 14, 2021

Depending on your CI/CD there are other tricks to speed it up since this doesn't do SPM yet

https://github.com/hendrikmuhs/ccache-action/ or

https://github.com/mikehardy/buildcache-action

Those are for github but the general solution design ("Install compiler cache tool, make symbolic links to the tool (or update your build definition to use it directly), save and restore compiler cache across builds") is generic and I have seen it work on bitrise as well so I know it's generally applicable.

@mikehardy do you have a bitrise.yaml example?

@mikehardy
Copy link
Contributor

mikehardy commented Aug 14, 2021

No, sorry. But I've heard of a successful bitrise implementation of ccache using xcodebuild directly on command line with variables LD/LD++/CLANG/CLANG++ set on the command line to the path of ccache

@Ahmadshoh
Copy link

Ahmadshoh commented Jan 17, 2022

Thanks. It works for me. True, I had to change the version from 7.14.0 to 8.10.0, if anyone has an error like that:

[!] CocoaPods could not find compatible versions for pod "FirebaseFirestore":
In Podfile:
FirebaseFirestore (from https://github.com/invertase/firestore-ios-sdk-frameworks.git, tag 8.10.0)

cloud_firestore (from `.symlinks/plugins/cloud_firestore/ios`) was resolved to 2.0.0, which depends on
  Firebase/Firestore (= 7.14.0) was resolved to 7.14.0, which depends on
    FirebaseFirestore (~> 7.14.0)

@Salakar Salakar closed this as completed Jan 28, 2022
@irajeshh
Copy link

irajeshh commented Feb 19, 2022

It took 1hr before & 5 mins after using this.

@mikehardy
Copy link
Contributor

mikehardy commented Feb 19, 2022

@irajeshh if you think that's neat (and it is!) imagine integrating ccache for your entire iOS build, so that everything (not just firestore) was only built when necessary...

@gvsakhil
Copy link

gvsakhil commented Feb 22, 2022

Omg this is scary my build times were more than 1hr and now it went to 10-20 secs...

I followed step 4 from this url -
https://firebase.flutter.dev/docs/firestore/overview/

@irajeshh
Copy link

irajeshh commented Feb 23, 2022

@irajeshh if you think that's neat (and it is!) imagine integrating ccache for your entire iOS build, so that everything (not just firestore) was only built when necessary...

Yeah it's okay for me, Also I just tested the macOs platform, Linux is my main developement base.

@firebase firebase locked and limited conversation to collaborators Feb 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests