Issues Cloning Spec repo - GitHub taking a very long time to download changes to the Specs Repo

[jlubeck]

Note from @orta -

If you are here because your Specs repo isn't updating, run: cd ~/.cocoapods/repos/master && git fetch --depth=2147483647 - this will convert your local repository of Podspecs to be a full clone, as opposed to a shallow copy.

---

What did you do?

Run pod setup

What did you expected to happen?

Clone Spec repo master

What happened instead?

It only downloads a few bytes and then throws error:

fatal: unable to access 'https://github.com/CocoaPods/Specs.git/': transfer closed with outstanding read data remaining

Podfile

No Podfile yet

I also tried cloning the repo manually or with the githhub desktop app with no avail.
I´m having no issues cloning any other repo in github. Only with this one. Is it possible there is something wrong with it???

Thanks

[jlubeck]

Tried again, new error:

error: RPC failed; result=18, HTTP code = 200
fatal: The remote end hung up unexpectedly
[!] /usr/bin/git clone https://github.com/CocoaPods/Specs.git master --depth=1

Cloning into 'master'...
error: RPC failed; result=18, HTTP code = 200
fatal: The remote end hung up unexpectedly

Very weird...

[ecch531]

I got same issue, too.

[mangofever]

Same.
Cannot clone spec repo.

[art-divin]

+1, same issue
git clone https://github.com/CocoaPods/Specs.git takes forever

[stringsanbu]

+1, been messing around with this for awhile. I doubled the buffer, didn't work. Uninstalled and reinstalled pods, didn't work. Tried to clone manually, no cigar. It actually seems to be getting "something" but fails. Using verbose didn't say much, just said it had issues accessing it.

I tried accessing my other repos and it seemed to be OK, but it was definitely slower than normal.

[huinme]

+1

I got same issue, too.
my pod version was "0.39.0"

I tried cloning master repos directly (by git clone git@github.com:CocoaPods/Specs.git master --depth=1 --verbose), but also failed.

[oronbz]

+1

[aceontech]

+1

[pedrocid]

+1

[MarkMolina]

+1. No success after increasing buffer / reinstall / manual clone

[stringsanbu]

Temporary workaround which might work: https://github.com/CocoaPods/Specs/archive/master.zip haven't tested though

wget https://github.com/CocoaPods/Specs/archive/master.zip

[samuel-mellert]

https://github.com/CocoaPods/Specs/archive/master.zip would be the correct link, I guess?

[fraxool]

Same error here. What should we do with the file at https://github.com/CocoaPods/Specs/archive/master.zip ?

[stringsanbu]

My bad, the wget link is correct. Just edited my first link.

[stringsanbu]

Not sure what to do with the file yet. Trying to see if we can manually run the commands to have pod setup work.

[aceontech]

Yeah, it merely downloads the repo's contents. The .git/ directory is missing, so it's not recognized as a git repo.

[samuel-mellert]

yes.. same here.. It always tries to clone the master repo. Even when I run it with --no-repo-update I get "Creating shallow clone of spec repo master-1 from https://github.com/CocoaPods/Specs.git"

[MarkMolina]

Did anyone try this with 1.0.0 beta 4?

[huinme]

@MarkMolina I tried, but same result.

[czechboy0]

Try to cd into ~/.cocoapods/repos/master, then git clean -fd to clean up the working copy, git checkout -- . to ensure you're on master and then git pull manually. This took ages but worked for me.

[SoundBlaster]

+1

[aceontech]

Thx, but I removed my master spec repo before I realized something was up with the Github repo ^.^

[stringsanbu]

Got a temp workaround! Tested with my app and everything is working. This is really only needed if you deleted the master repo. If the master folder is still in your ~/.cocoapods/repos folder with contents then you should be ok to just use pod install --no-repo-update.

• Try doing a pod setup. This should at minimum download the .git to ~/.cocoapods/repos/master
• While this is going, you need to move the .git folder from the master folder to somewhere temporary.
• Stop the pod setup. Delete master folder in repos
• Use the wget command below to get the zip of the repo
  wget https://github.com/CocoaPods/Specs/archive/master.zip
• Unzip the master zip and move its contents to ~/.cocoapods/repos/master
• Move your .git folder from wherever you put it to ~/.cocoapods/repos/master as .git
• Go to your project folder, do a pod install --no-repo-update

And you should be good to go!

So in short, here is the basic list of commands I used:

pod setup (in a separate tab)
mv ~/.cocoapods/repos/master/.git ~/tempSpecsGitFolder
^C on pod setup tab
wget https://github.com/CocoaPods/Specs/archive/master.zip
open master.zip (unzipping it)
mv Specs-master ~/.cocoapods/repos/master
mv ~/tempSpecsGitFolder ~/.cocoapods/repos/master/.git
cd [project folder]
pod install --no-repo-update

[aceontech]

Is this a Cocoapods or a wider GitHub issue?

[stringsanbu]

@aceontech Pretty sure it is a GitHub issue, but my other repos are working fine so perhaps only certain repos on certain servers (on their backend) are affected.

[aceontech]

I was just able to do a successful pod setup. Don't know if it's repeatable.

AlexMacBookPro:repos alex$ pod setup --verbose

Setting up CocoaPods master repo

Creating shallow clone of spec repo `master` from `https://github.com/CocoaPods/Specs.git` (branch `master`)
  $ /usr/bin/git clone https://github.com/CocoaPods/Specs.git master --depth=1
  Cloning into 'master'...
  Checking out files: 100% (74393/74393), done.
  $ /usr/bin/git checkout master
  Already on 'master'
  Your branch is up-to-date with 'origin/master'.

[SoundBlaster]

Github is very very slow: ~ 40-50KB/s

[pedrocid]

I was able to do pod setup -- verbose right now.

[segiddins]

This is a GitHub issue rather than a cocoapods issue -- you're best off reporting it to their support rather than us, since there's nothing we can do about it.

[tychop]

And github refers to Cocoapods... Great...

[solarce]

Hey there, I am the infrastructure manager for Travis CI and my team is responsible for our OSX build images, which includes CocoaPods and the speed of the pod setup operation in each build is something we've been discussing lately.

We'd definitely love to hear more about we how could:

1. Improve how we do an initial install and setup of cocoapods and the specs, when we build new images for new Xcode updates, e.g. 7.3b5 yesterday. Right now we just do https://github.com/travis-infrastructure/osx-image-bootstrap/blob/master/bootstrap.sh#L73-L75
2. If there are some improvements we can make to the things we do at runtime, in https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/script/objective_c.rb and/or how our build in caching handles the specs

[codeOfRobin]

Stupid Question. Are similar load problems seen in repos like Homebrew(which maintains similar kinds of information) as CocoaPods?

[ZevEisenberg]

I'd like to put in a plug for checking your Pods folder in to source control. I know there are arguments on both sides, but I've been checking in Pods for over a year on all projects and loving it. It removes your project's dependence on pod install in order to run on CI, which speeds up builds, and the repo size hit isn't really a big problem in my experience. One note: you'll generally want to change your Podfile and run pod install and merge that separately from any other changes, to keep your pull requests readable.

[jcampbell05]

@solarce Aside from using Travis CI caching (which is awesome BTW) and the ideas above (Which are a positive direction and great work on the quick response from @alloy and the team ). What I've been using as a solution for my company is a plugin I wrote which attempts to use the bundle install --deployment functionality.

Check it out here:
https://github.com/jcampbell05/cocoapods-deploy

It looks at the lockfile, and uses the raw HTTP links to the podspecs for each dependency and downloads them. This means that we don't need to clone the CocoaPods Specs repository at all, we can take advantage of Github's HTTP caching (Reducing the workload for @vmg and @mhagger) and TravisCI can cache these Podspecs as well (Since they reside in the Pods folder).

With this plugin we've experienced a 85% decrease in CocoaPods install times (Down from around 500 seconds to 85). I would love to know your thoughts on this and to see if it works for you.

[jcampbell05]

@ZevEisenberg I've never been a fan of that approach and always felt it could be something that should be solved via a CocoaPods's plugin (like the one above). Since other package managers solve this via a special deploy mode which doesn't require you to mix your dependencies and source together.

I think its important to give people the choice :)

[orta]

OK, looks like it's time to lock this thread. Sorry people wanting to continue this conversation productively, but we've hit the hackernews/mainstream point where the tone of the conversation is going awry due to outsiders. If you'd like to reach out to me or the CocoaPods team, you can direct message me on twitter (same username) for a quick response, or use info@cocoapods.org for something team-y.

GitHub staff we totally appreciate the discussion, and will update this thread once we've got somewhere towards having a release with some fixes. I'll unlock it then 👍.

[alloy]

Shame :(

[segiddins]

Just a quick update: I've just released CocoaPods v1.0.0.beta.6 with a few improvements to how we handle specs repos, particularly https://github.com/CocoaPods/Specs. The full CHANGELOG is available at https://github.com/CocoaPods/CocoaPods/releases/tag/1.0.0.beta.6, but in short:

• Specs repos will no longer auto-update by default when running pod install
• When attempting to pull the master specs repo, we check that preview API for a 304 and will not perform the fetch if the API says we have the latest commit
• Shallow clones of specs repos are no longer supported
• Shallow clones of the master specs repo will be automatically unshallowed.

In the next beta, we will be adding forward compatibility support for a sharded specs repo directory structure, to be migrated to in a few months.

We hope that this set of changes will lead to a better user experience in the long run for all users of CocoaPods, and will also help ease the burden of serving the master specs repository that GitHub faces. I speak on behalf of the team when I say thank you to everyone who has helped us over the past week, in particular @DanielTomlinson and @mrackwitz for making massive pull requests, @orta and @alloy for helping coordinate this effort, and the entire GitHub team, particularly @vmg, @mhagger, and @arthurnn for providing technical advice and infrastructure to CocoaPods, our users, and the entire open source community.

If anyone has further questions / comments about this release and the work to come, please feel free to reach out either to me personally (segiddins@segiddins.me) or to the CocoaPods team (info@cocoapods.org). Thank you everyone for your patience over the past week.

-- ❤️ The CocoaPods Team

[kylef]

I'd suspect it might be a while for users to update to 1.0.0 considering it includes many breaking changes over previous versions. Does anyone think it would be worth back porting these fixes to the 0.39.x series and issuing a patch update removing shallow etc?

Especially if the master repository is going to be sharded and become incompatible with 0.39.

[jcampbell05]

@kylef I agree it would be good to have a 0.40 with some of these changes.

[vinnybad]

@kylef it would certainly help our teams!

[segiddins]

Thanks everyone for sticking with us! We've written up a post-mortem of the issue and the steps taken to mitigate at http://blog.cocoapods.org/Master-Spec-Repo-Rate-Limiting-Post-Mortem/.

[dnkoutso]

Please see https://blog.cocoapods.org/CocoaPods-1.7.2/ as version 1.7.2+ has switched to use CDN and fresh installs of CocoaPods do not need to clone the master specs repo anymore.