Remove Git dependency from bundler #1043

Closed
hedgehog opened this Issue Feb 15, 2011 · 15 comments

Projects

None yet

8 participants

@hedgehog

bundler.gemspec requires:

  • git
  • bundler sources be in a git repo

This precludes distributing bundler source as tar balls, etc., e.g.:

wget --quiet --no-check-certificate -O bundler.tgz https://github.com/carlhuda/bundler/tarball/<tag>

rake install fails 'silently' and then comes the boom:

$ irb -rubygems
ruby-1.9.2-p136 :001 > require 'bundler'
LoadError: no such file to load -- bundler
        from <internal:lib/rubygems/custom_require>:29:in `require'
        from <internal:lib/rubygems/custom_require>:29:in `require'
        from (irb):1
    from /home/hedge/.rvm/rubies/ruby-1.9.2-p136/bin/irb:16:in `<main>'

Now you have to work out that to install Bundler requires both Git be installed and sources distributed by Git....

@benhamill

So... Maybe I'm misunderstanding you completely, but this seems like a very strange thing to ask for.

The official bundler repository is on GitHub and, thus, git is at least an implicit dev dependency for working on bundler if you're concerned with either keeping pace with or contributing back to official git releases. So having the gemspec depend on git doesn't ADD a dependency.

If you're not concerned with official bundler, then you're free to ditch git, but then why do you care what is in the official bundler gemspec? You can change the gemspec to whatever you want and version control with mercurial, svn or nothing at all, if you want and distribute it however you want.

I assume that the version of bundler you're passing around is hacked in some way or else why pass around compressed source and not just .gem files, which should work fine without git, no?

@benhamill

Let me ask a more basic question: Why are you wanting to pass around source in this way? If it's unchanged, pass around the .gem file, no? Don't build it, get it from rubygems.org, then you don't need source at all.

@benhamill

I don't know that it's VERY problematic. But it seems like since you need git if you want to interact with bundler's official source repository, then it's not hurting anyone to be there. If you want to do something with the source after that means it doesn't have to rejoin the official bundler source, you can change the gemspec to whatever you want.

I don't think I understand your second question, so I can't answer.

What would you get from this change being in bundler official that you couldn't get from forking this repo, changing the gemspec as you say and then keeping it up to date with future bundler releases? You could build the .gem files in your own to distribute as you wish, or distribute the source of your own repo, zip up the code with the .git directory removed and build it on some machine that doesn't have git installed, if you wanted, no?

@benhamill

I want to point something out: I don't have commit access to this repo. So I'm not a person who can decide whether a pull-request with your idea would be accepted or not. I'm just trying to understand what you're proposing and why. Sorry if that wasn't obvious up front.

I don't know how I misunderstood this before, but I just realized that when you were saying "download" that wasn't just a generalized term for cloning the repository. You meant literally the button marked "download". Also, I didn't realize that those downloads were absent the .git directory and so weren't, themselves, already repositories.

So I had in my head that idea that you were wanting to spend the time to remove the .git dir for some reason and pass the source around like that. That idea seemed insane. So I started asking questions about it. NOW I understand what it is you're actually outlining and the drawback you're describing.

The benefit, I guess (having not written the code myself) is that if you're hacking on the bundler source and have files sitting around your file tree (like editor-created project files or whatever), git would be aware not to list them (assuming you'd ignored them properly) and so the automated list is a little more robust than what you might build in Ruby to do much the same thing: look for files automatically, rather than have to hand-maintain a list of all the files in such a big project.

As it stands, if you had cloned the project, compressed the file and put it on a machine with not net access, but with git installed, it would build from source just fine, right? Or am I still missing something? Also, is there a reason that building from source (which, in your example, would be stored in a disaster directory) is better than having the .gem file in your disaster directory?

@hedgehog

I want to point something out: I don't have commit access to this repo. So I'm not a person who can decide whether a pull-request with your idea would be accepted or not. I'm just trying to understand what you're proposing and why. Sorry if that wasn't obvious up front.

It wasn't and it is my fault for not asking specifically.

@rodrigues

Our team begun recently to build debian packages for our apps. Bundler helps a lot to make our .deb files self contained, have all needed gems, bundle install --deployment is the best, thanks.

However, since bundler doesn't put itself into vendor/bundle, we need to pack bundler.gem in the debian file using gem unpack + gem build.

Having git commands to list files in .gemspec make this work kinda harder. When you unpack bundler, it doesn't come with .git files, and we hack the folder with git init && git add -A to be able building the .gem.

For me sounds very reasonable to change this, because I don't see a concrete benefit of using git ls-files instead of a Dir[exp].

@kikito

Hi,

I hope it's ok if I try to contribute to this issue (which is old, but still opened).

6 months after the last comment Tenderlove wrote a post about profiling Rails' startup time. He pointed out that one of the culprints was that some gemfiles where shelling out and executing git ls-files. It turns out that rails spent roughly 11% of its bootup time was spent on such subshells.

The post in question: http://tenderlovemaking.com/2011/12/05/profiling-rails-startup-with-dtrace/

If I've understood this correctly, we should consider removing the git shellout not only because of an "extra external dependency", but for performance reasons - apparently they "add up" when used in lots of gems in a rails project (which is a typical scenario).

Now, I'm not as good as Aaron on profiling. If I understood correctly, the time is spent shelling in/out, so replacing the git command by a Dir[*/] action should take care of that:

gem.files = Dir['**/*'].select{|f| File.file?(f)}

The results of this expression are quite similar to the ones obtained with the shellout, with the exception that all files begining with a dot are ignored. In a typical gem, the only file matching this description is .gitignore, so this should be a pretty safe value.

Strangely Aaron doesn't mention anything similar on his post, so maybe I'm missing something.

@matschaffer

+1 on not shelling out from a gemspec. I'd like to at least see a switch to use a file glob instead.

@mscottford

After reading the discussion in #2023, it looks like this issue should be closed. Any objections?

@matschaffer
@matschaffer

So I made a test which contains two slow gems (sleeps during file listing) and a rails app. I used the time to run rake as a measuring tool to spot the slowdown. A few interesting notes

  • installed gems (using rake install) don't have a shell-out performance penalty
  • path-included gems (using `gem '...', path: '...') do have a penalty
  • the penalty isn't paid only once: including a 10 second sleep caused a 40 second increase in time to run rake

I suspect git-included gems will also have the penalty.

I've been pretty happy with what I do in https://github.com/matschaffer/knife-solo/blob/020824b5daa24ba8169717f06dbf1408725c7f65/Rakefile#L4-L24 and https://github.com/matschaffer/knife-solo/blob/48757d1ff1751b31a7beb3d4eeffe4fe28bc0492/knife-solo.gemspec#L13-L16

It's basically hoe-inspired but implemented on top of bundler/gem_tasks. If there's interest I can work on a PR.

@matschaffer

Oh, the test repo is at https://github.com/matschaffer/gemspec_shellout_test for the above findings.

@matschaffer matschaffer added a commit to matschaffer/bundler that referenced this issue Jan 29, 2013
@matschaffer matschaffer Use Manifest.txt and .gemignore based gemspec listing #1043 0a8e7b1
@matschaffer matschaffer added a commit to matschaffer/bundler that referenced this issue Jan 29, 2013
@matschaffer matschaffer manifest rake helper #1043 8f8b207
@matschaffer matschaffer added a commit to matschaffer/bundler that referenced this issue Jan 29, 2013
@matschaffer matschaffer Changelog proposal for #1043 ce3afe3
@matschaffer

Bah! who needs interest? :)

@xaviershay

Closing coz it's a feature not a bug and we're using this tracker for bugs. Please do re-open in bundler features if you would like to continue discussion (or with a different PR - I see the last one was not merged.)

@xaviershay xaviershay closed this Aug 10, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment