Enable direct HTTP repo deployment and update of pods

[jcampbell05]

• I've read and understood the *CONTRIBUTING guidelines and have done my best effort to follow.

Overview

It Should be possible to directly install (or "deploy") and upgrade cocoapods from a repo using HTTP.

Why it is Important?

To Quote @art-divin
"Recently there was an issue with some weird GitHub outage. The same goes for bad networks etc. This has caused many issues with running pod setup, pod install, pod update etc. for a number of developers.

Problem was not related to Podfile configuration and did not had to do with anything configuration-related. It was specifically about git clone https://github.com/CocoaPods/Specs.git failing 1 out of 10 times for whatever reason (bad network, service outage etc.)"

In addition to this. Cocoapods experiences long install times on CI Systems such as Travis where it has to pull down the entire repository for each build.

Expected Outcome

As a CocoaPods developer I would like CocoaPods to be able to update and install without having to pull down the entire CocoaPod Specs Repo

Notes

• I have already built a plugin for the pod install functionality to be ran directly from HTTP requests in a plugin called CococaPods-Deploy https://github.com/jcampbell05/cocoapods-deploy

It worked by analysing the lockfile, downloading the external sources for all pods and dependencies and downloading the podspecs for the version of each pod and dependencies. It would then patch the system in such a way that when it detected the system was using a Dependency from a Repo it would use the podspecs it had just downloaded and prevent the Installer from writing to the lockfile.

I also overrode the repo sources as the skip flag doesn't stop cocoapods cloning the repo sources too.

With our CI we've experienced (in combination with Travis CI Pods caching) an 85% speed up with this, since it can just directly install the dependencies.

• For pod update we could in theory do something similar, if we had some kind of manifest file in the root of each library folder in a repo that listed the versions. We could download this via HTTP when people specify it in their podfile. Then we instantly know all the available versions without pulling the whole repo. Bundler in effect does the same thing with a API but we are using Githubs infastructure here.
• We should update the lockfile to contain information about from what repo a pod was downloaded from (Gem lockfiles have this information which is used to speed things up).

[orta]

We've tried this in the past, and eventually decided against it.

Simplicity is more important, using normal day to day dev tools to perform tasks ( e.g. using git clone or curl instead of trying to recreate everything ourselves. Having the whole specs repo locally instead of doing rubygems style server-resolving.

Yes, GitHub's cloning is being unreliable right now, but their API can be just as bad.

[jcampbell05]

@orta for me given the fact we are now rate limited #4989 (comment)

Bundling a deploy function would help a great deal. You would only need to clone the full repo when adding a new dependency or updating an existing one. deploy would be used for the majority of the cases (especially for CI) where all you want is to install the dependencies based on the resolved version numbers in the lockfile. Cloning the repo is only really useful when you want to know about potential new version (like in the case of pod install and pod update).

The remaining issues with git being slow could be solved like mentioned in #4989 (comment) by removing the --shallow option, hopefully this would live on a developer's machine so would be a one time cost to get setup with CocoaPods and because the developer only needs it for new dependencies or updating existing ones.

deploy should reduce the need for the developer to wait for their repo to be updated each time they need to update to the version requirements in the latest lockfile and reduces the overall load on the Specs Repo (Which are probably made worse with people using CI's such as Travis). Since if all they are needing to do is make sure they are using the correct library version for the project then we can do this without the repo clone (Which as I mention is a big help for CI and strain on Github).

This could be a good solution until if and when we get a better one. I have a working POC in my plugin I wrote for a deploy function here: https://github.com/jcampbell05/cocoapods-deploy

[alloy]

The best code is no code. As such, it would be better if we can fix the current design, rather than accepting defeat early and go off and build more tools.

[jcampbell05]

@alloy true, I'll see how things pan out. However in any case, I still feel there is a case to bundle a deploy functionality some time in the future (as a plugin).

[segiddins]

The plugin already exists. I don't even see what this issue is asking for, given --no-repo-update exists.

[jcampbell05]

@segiddins the issue with --no-repo-update is the repo is still updated if a pod points to it.