Add: Support for shallow clones

[jdevera]

This is a feature request.

It'd would be convenient to have a user settable global variable that controlled whether the git clone command that is execute to download plugins makes a full clone or the original repo or a shallow copy.

Ideally, this shallow copy would include only the latest revision (--depth 1)

It would be even nicer if whatever the global setting is, it could be overridden on a per bundle basis.

Shallow clones can speed up download time significantly if there are many plugins installed, specially when repos have many revisions.

[randymorris]

I'd just like to add a +1 from me. . I actually think this would probably even be a sane default. You could most likely use the same mechinism as the custom naming uses now to override this option.

[0]

I think this does the trick.

Note that since all the repositories in vim-scripts have a tag for every commit, this change has no effect for them. I was unable to find a way to not clone the commits which have a tag associated with them.

[jdevera]

I think having a simple switch as configuration is a simpler approach that would cover practically all cases. I can't imagine why someone would want any depth other than 1 or the full thing, so I'd suggest something like
let g:vundle_shallow_clones = 1 to set the depth to 1 and a default 0 to clone the full repo.

[randymorris]

@jdevera that wouldn't allow you to fully clone specific repos while shallow cloning others. @0's approach is flexible without being too complex.

[jdevera]

@rson, I probably did not explain what I meant correctly, what I meant is to basically keep the same logic @0 has put in place, only change the control variable name and meaning, from g:vundle_depth which is passed directly as the --depth parameter for git, I suggest having only depth 1 or the whole repo, and have this controlled with a more generic variable name, could be g:vundle_shallow_clone or we could even make shallow clones by default and override with g:vundle_clone_history.

This being a global setting doesn't mean that it cannot be overridden on a per plugin basis, with something like {'history' : 1}

[0]

@jdevera, I feel that in your case, you lose generality with no loss of complexity. With my version, you can specify a global or local depth of 1 to do the shallowest clone, or of 0 to do the deepest clone. Or if for some reason you so fancy (but only if you do), any other depth value to your liking. With your version, you could do the first part (in the exact same manner, but with different names), but not the second.

That said, it's ultimately up to @gmarik, so for the time being I won't make any changes to the code I pushed.

[gmarik]

wow, what's going on here.... )

There are not many options, which makes Vundle simple, which is important, IMHO
And there weren't any complaints like "Vundle is slooooow please fix it"...
So I guess this is low priority feature for now...

But the part with using global variables/naming is kinda important:

what if instead using global variables we'd just pass options:

call vundle#rc({'clone':{'depth': 0}})

What do u say?

[jdevera]

Well, I don't really mind as long as it works :), I guess that kind of nested structure looks more awkward than a global variable, which is a very common method of configuring plugins in vim. However, I am guessing that because vundle has to be called at the beginning of vimrc, it already is a special case and thus the special type of tuning method makes sense.

But really, at the end of the day, what matters is that it works, I don't think a lot of people will use this feature, and those of us who would use it probably don't mind either way.

[gmarik]

yeah, i agree, hashes aren't as readable as global variables...
Maybe : g:vundle#clone#depth = 0 or g:vundle_clone_depth = 0 vs g:vundle_depth ?
Naming is hard! )

[jdevera]

Your second option (let g:vundle_clone_depth = 1) seems more descriptive and less awkward looking to me. I'm guessing the default value would be 0 anyway, right?

[0]

@gmarik, is there anything you disagree with in my pull request other than the naming?

[gmarik]

@0 if your question is why pull request haven't been merged then the answer is that ppl had lots of issues with current features/documentation. Which is a sign that Vundle needs better test suite before adding new features. So it's kind of "feature freeze" for now.
Adding features is fine as long as we are consistent and not breaking anything.

Let me know what you think...

[ches]

I don't really have an opinion on how this should be configured, but IMO a clone depth of 1 should be default. The vast majority of the time people are going to be installing bundles as plug-and-play, saving space and time with shallow clones is a no-brainer. If you want to hack on a particular plugin and contribute upstream and/or push customizations to your own fork, it's trivial to convert a shallow clone to a full one on an as-needed basis.

[gmarik]

That's true, but there's another side:
There's a feature request to be able to specify revision, combining it with default depth:
Bundle 'rails.vim', {'v':'v0.8', 'depth':1}

But don't they conflict with each other?

[0]

@gmarik, the shallowest possible clone gets all branch tips, all the tagged commits, and all their parents, so it would still always be possible to refer to named (branch/tag) commits that way.

Referring to an individual commit by its hash would obviously not work if that commit doesn't exist, but I think this is reasonable for the general case. On the other hand, if you're going through the trouble of specifying a commit hash, you can also disable shallow cloning.

[Taverius]

@gmarik, you're confusing me, 'v': will work for tags as well as revisions?

I see what you mean though. I mean, you could make 'v' disable shallow but that only works if 'v' is specced on clone ...

... if one specced a hash on a shallow clone you'd have to trash the repo and re-clone, or error-out descriptively.

On the other hand, it makes it easier to make an 'I want only tagged revisions' update mode, shallow-clone, checkout the latest tag, and check for a newer tag on pull. That would be handy, since understandably some people don't push every minor version to vim.org

[gmarik]

@Taverius yes, 'v' should work for both

@0, cool, didn't know that!

The use-case i'm thinking of looks like this:

1. user clones bundles(with depth=1 by default)
2. decides to go to specific revision by setting {'v': 'abcdef123'} (ie when master is broken)
3. vundle errors out as the revision cannot be checked out due to (1)

Now how this situation should be dealt with?
4) Vundle figures out that as repo was cloned with --depth=1 then specified revision might exist in full clone...
Will Vundle be responsible for re-cloning the repo?
5) reclone repo and try checking out the revision
What if that revison doesn't exist ?
6) error out, invalid revision. quit trying

IMO above looks like a lot of state-management for Vundle, will make code way more complicated...

[jdevera]

It's been a while since I opened this feature request and during this time I've been cloning the full repositories for all my increasing number of plugins. To be honest, I think I should not have opened this issue. I said:

Shallow clones can speed up download time significantly if there
are many plugins installed, specially when repos have many revisions.

I no longer believe that the word significantly is well suited here. Git compresses its repositories really well and also most of the plugins we are going to be getting with Vundle come from the vim-scripts user, which only creates one commit per uploaded version in the official site. So most of the time there won't be many revisions and most of the time, the size is going to be really small (just a couple of text files).

I believe Vundle should be kept simple and that being able to specify a commit/tag/branch is far more useful as a feature than having shallow clones. If having both means it's too easy to screw up the config or that Vundle needs to start checking a thousand conditions and having nice bailouts for all, I'd say drop the depth feature instead.

[gmarik]

@jdevera good points.
Adding more features will definitely make Vundle harder to use.

[gmarik]

(I thought I posted this comment, but seems like I cannot find it anymore, so reposting again):

Unrelated to this issue:
nin branch pushed which has new installer. If someone can find couple minutes to check it out (both literally and not) it'd be very helpful!

cd vundle; git pull && git checkout -b nin origin/nin
start vim and run :BundleInstall
Watch "awesome installer" in action )

Let me know what you think.
Thanks!

[jdevera]

I tried it, but I get errors (I assume a red exclamation mark means an error) and could not see what the real problem was. Seems like error messages are swallowed by vim. Is there any way to see error messages?

[gmarik]

yes, red exclamation mark means errors.
run :VundleLog right after installation finishes. It's not perfectly detailed but i hope you'll be able to get an idea.
What bundle errors out?

[jdevera]

I've created a new issue (#69) to track it. All details there.

[gmarik]

@jdevera - thanks for the report!

[upsuper]

@jdevera @gmarik Sorry for digging out this issue, but after reading the discussion above, I still think this feature is necessary.

Today I installed rust support via Vundle from Mozilla's official GitHub repo by adding the following line to my .vimrc:

Bundle 'mozilla/rust', {'rtp': 'src/etc/vim/'}

There was a long time before it finished downloading, because it downloaded the whole rust repo with all its submodules, which occupied nearly 500MB in all. After I removed the llvm submodule manually, which was 187MB, total size of the repo became 314MB whereas the .git directory itself wastes 292MB, more than 10x larger than the code of rust, and over 10,000x larger than the vim plugin I need actually which is only 20KB or so.

I know that most vim plugins are in their own repoes, which are not so large. But there are some exceptions like this case, indeed. Anyway, I don't think downloading and storing such huge useless data is a good idea, so this feature is still desirable.

[jdevera]

@upsuper It's fine, the issue is still open anyway. I'd still like to see this happening in Vundle, although I never thought of the rpt parameter case where the overall repo is huge. What I mean is, that we'll eventually get there.

I think for this particular case maybe having a shallow clone would perhaps not save you enough space/bandwidth. I had a similar issue once where vim support was part of the product and ended up creating a separate repo for the vim files.

Alternative, you can ask the maintainer to provide the vim support files as a separate repo, so everyone wins.

[e0da]

👍 This would save a lot of people a lot of time. To quote @gmarik himself: "Vundle is slooooow please fix it" 😜

[Shougo]

No. Vundle should not support shallow clones.

Shougo/neobundle.vim#81

See this issue problem.
Shallow clones does not save your time and has some problems.

[MarcWeber]

also related: #335, #29

[e0da]

@Shougo I don't understand the assertion that it does not save time. It absolutely does. But clearly it's not a viable solution if it's going to break with a significant number of plugins. Vundle is just going to be throttled by git. 😢 That's OK. Certainly not a huge deal. ❤️ Vundle!

Thanks @Shougo and @MarcWeber for the additional information. I appreciate it.

[jdevera]

Perhaps we could, as @gmarik suggested years ago, configure this per Bundle. So you can configure it only on bundles that can handle it.

[e0da]

@jdevera 👍

[Shougo]

Perhaps we could, as @gmarik suggested years ago, configure this per Bundle. So you can configure it only on bundles that can handle it.

OK. I think it is better solution.

[MarcWeber]

VAM has been supporting shallow clones for a long time without issues.

[andrewhamon]

I'd like to chime in that for some of us time isn't the issue, but space.

I sometimes have to ssh in to a computer at school to submit assignments, and my disk quota is 15MB. I can not install all my normal plugins :(

[jcppkkk]

Vundle should support shallow clone per plugin basis

For example, if i install docker syntax with setting
Plugin 'docker/docker' , {'rtp': '/contrib/syntax/vim/'},
This syntax plugin will cost me 156 seconds and 103MB disk space to install... 😵
Sometimes full cloning plugin can become a burden.

[bijanbina]

Such a nice feature but it seems sadly developer aren't going to add this :(

[Shougo]

@jcppkkk

It seems docker project's problem.
Docker should split the syntax file from it.