Better handling of large repositories when used as a git gem.

[coilysiren]

From @xaviershay on August 7, 2013 2:44

Using git for large gems is slow to clone and uses a lot of disk space. Moved from rubygems/bundler#228

Approaches discussed include:

• Using a shallow clone (--depth 1). Doesn't work for fetching specific refs, which happens when loading from a Gemfile.lock, even if you don't specify the ref option in your Gemfile.
• Solving the problem elsewhere (such as by excluding the git directory when compiling a slug for heroku).

Copied from original issue: rubygems/bundler-features#1

[coilysiren]

From @elgalu on November 25, 2013 20:59

+1 for allowing the option depth: 1

[coilysiren]

From @indirect on November 25, 2013 23:45

How do you even allow it, though? Your lock will always have a specific ref, and --depth 1 forces you to use the newest commit in the repo. Those are mutually incompatible.

[coilysiren]

From @TimMoore on November 26, 2013 0:6

In the case of a forked gem, it seems like a pretty common case that the newest commit in the repo is the one you want to use.

I think it might be reasonable to allow a shallow clone option and simply fail if it doesn't match the locked ref. It can print a message saying to either bundle update the git gem or remove the shallow option.

[coilysiren]

From @indirect on November 26, 2013 0:8

Hmmm I guess that's true. Pretty sure that --depth 1 also only works with the master branch, but I am open to a patch for this. We'll just need to be very careful about documenting that this only works for repos that are not under active development.

[coilysiren]

From @elgalu on November 26, 2013 0:20

I've done some research and it seems there is a way to download an specific ref zipball / tarball

This won't download git revisions nor history, just the git blobs (files) of some specific ref

# e.g. get Rails 4.0.1 expanded tarball
time curl -L https://github.com/rails/rails/tarball/5505c1d700 | tar zx
# real  0m17.263s
# user  0m0.584s
# sys   0m0.360s

# compare git clone (even though this doesn't solve the current feature)
time git clone https://github.com/rails/rails.git --depth=1
# real  0m23.989s
# user  0m1.524s
# sys   0m0.396s

[coilysiren]

From @indirect on November 26, 2013 0:28

Uhh… you realize that there are many Bundler users getting git repos from places other than Github, right?

On Nov 25, 2013, at 4:20 PM, Leo Gallucci notifications@github.com wrote:

I've done some research and it seems there is a way to download an specific ref zipball / tarball

This won't download git revisions nor history, just the git blobs (files) of some specific ref

e.g. get Rails 4.0.1 expanded tarball

time curl -L https://github.com/rails/rails/tarball/5505c1d700 | tar zx

real 0m17.263s

user 0m0.584s

sys 0m0.360s

compare git clone (even though this doesn't solve the current feature)

time git clone https://github.com/rails/rails.git --depth=1

real 0m23.989s

user 0m1.524s

sys 0m0.396s

—
Reply to this email directly or view it on GitHub.

[coilysiren]

From @elgalu on November 26, 2013 0:38

Yes!!

But maybe the great majority are Github users, i'm trying to get some scm usage statistics correlated to bundler, will let know if i get the number.

What if >= 80% are bundler+Github users? Would the feature be worth building, i imagine something like this:

source 'https://rubygems.org'

gem 'rails', github: 'rails/rails', ref: '5505c1d', tarball_only: true

[coilysiren]

From @indirect on November 26, 2013 1:4

I mean, you could build it in as a way to make a shallow option that works only in concert with a github option (or git urls that point at github). Alternately, the shallow option only allows master with no ref specified, I guess.

On Nov 25, 2013, at 4:38 PM, Leo Gallucci notifications@github.com wrote:

Yes!!

But maybe the great majority are Github users, i'm trying to get some scm usage statistics correlated to bundler, will let know if i get the number.

What if >= 80% are bundler+Github users? Would the feature be worth building, i imagine something like this:

source 'https://rubygems.org'

gem 'rails', github: 'rails/rails', ref: '5505c1d', tarball_only: true
—
Reply to this email directly or view it on GitHub.

[coilysiren]

From @TimMoore on November 26, 2013 1:19

git clone has a --branch option:

--branch
-b

Instead of pointing the newly created HEAD to the branch pointed to by the cloned repository's HEAD, point to branch instead. In a non-bare repository, this is the branch that will be checked out. --branch can also take tags and detaches the HEAD at that commit in the resulting repository.

This actually sounds pretty useful even without shallow clones.

Also:

--[no-]single-branch

Clone only the history leading to the tip of a single branch, either specified by the --branch option or the primary branch remote's HEAD points at. When creating a shallow clone with the --depth option, this is the default, unless --no-single-branch is given to fetch the histories near the tips of all branches. Further fetches into the resulting repository will only update the remote-tracking branch for the branch this option was used for the initial cloning. If the HEAD at the remote did not point at any branch when --single-branch clone was made, no remote-tracking branch is created.

[coilysiren]

From @myronmarston on March 12, 2014 7:4

As @TimMoore pointed out, --depth can work in conjunction with --branch. The git docs mention that --branch can be given a tag instead of a branch and that works. For example, I tried this and it worked fine:

$ git clone git://github.com/rails/rails --depth 1 --branch v4.1.0.rc1

So...maybe bundler could have an optimization that if a tag has been provided, it'll do the shallow clone?

[coilysiren]

From @indirect on March 12, 2014 7:6

The tricky part is how to handle that branch or ref changing. What if it's a force push and the remote commits are now gone? How do you do an update? You have to do a completely new shallow clone. That slows down install and update by a huuuuge amount in exchange for saving some disk space, which doesn't seem worth it to me.

[coilysiren]

From @myronmarston on March 12, 2014 7:10

Well, I suggested doing it only for tags because tags are generally intended to be immutable. I wouldn't do it for branches or other refs. I suppose that since tags can still be force pushed over it might still not be doable, though.

[coilysiren]

From @myronmarston on March 12, 2014 7:10

Seems like git really needs to support a ref option :(.

[coilysiren]

From @indirect on March 12, 2014 7:17

Oh yeah, also that. If you can’t clone directly to a ref (which git doesn’t let you do) you’re basically screwed before you even start to try to deal with the problems of changing refs and updating meaning cloning 100% of the repo again. :/

On Mar 12, 2014, at 12:10 AM, Myron Marston notifications@github.com wrote:

Seems like git really needs to support a ref option :(.

—
Reply to this email directly or view it on GitHub.

[coilysiren]

From @sdhull on May 8, 2014 19:8

OK I recently ran into this issue for a project I'm working on, and I'd like to work on a PR for it.

Can we agree that if a user is using the github option, then we can leverage the tarball / zipball download feature of github, and otherwise we fallback to whatever the current behavior is?