Debugging of an old bug: Bundler failed to load gems which are :git or :github based and are installed system-wide

[nengxu]

This is an old bug that has been discussed many times before, including some tickets here. One example is at:
http://www.ruby-forum.com/topic/213962

The problem is:
If a gem is based on :git / :github, eg.:
gem 'dolphin', github: "nengxu/dolphin", ref: "60e2e753edd6fd655001f14b065c2db386c43cf3"

And it is installed system-wide, eg.:
sudo bundle install
sudo bundle show dolphin
/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/bundler/gems/dolphin-60e2e753edd6

When run 'bundle check' as App user (running puma / passenger, etc.), not as root, it will fail to recognize the system-wise installed gem:
bundle check
git://github.com/nengxu/dolphin.git (at 60e2e75) is not checked out. Please run bundle install

Previously general solution to this bug is to run 'bundle install --deployment'. However, this approach is just to avoid this problem, not to solve it.

I've done debugging on this annoying bug. And found it is due to the path setting of git-based gem:

/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/gems/bundler-1.3.5/lib/bundler/source/path.rb:112:in `load_spec_files'

    expanded_path = path.expand_path(Bundler.root)

raise expanded_path.inspect 

will give:
/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/gems/bundler-1.3.5/lib/bundler/source/path.rb:113:in `load_spec_files': #Pathname:/home/neng/.bundler/ruby/2.0.0/dolphin-60e2e753edd6 (RuntimeError)

Obviously, Bundle only looks for the per-user bundle folder, not the system-wide folder of "/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/bundler/gems"

No wonder later:
/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/gems/bundler-1.3.5/lib/bundler/source/path.rb:143:in `load_spec_files'

      raise PathError, "The path `#{expanded_path}` does not exist."

which will caught by git.rb in line 176, and throw that exception we saw above:

  def load_spec_files
    super
  rescue PathError, GitError
    raise GitError, "#{to_s} is not checked out. Please run `bundle install`"
  end

Please correct this bug by searching also in system-wide bundler folder for git-based gems. Thanks.

[indirect]

sudo bundle install is not supported. please don't use it.

​loading git gems from GEM_HOME seems like a good idea, but that already happens if you don't use sudo to run bundle.

On Wed, Jun 5, 2013 at 3:51 PM, nengxu notifications@github.com wrote:

This is an old bug that has been discussed many times before, including some tickets here. One example is at:
http://www.ruby-forum.com/topic/213962
The problem is:
If a gem is based on :git / :github, eg.:
gem 'dolphin', github: "nengxu/dolphin", ref: "60e2e753edd6fd655001f14b065c2db386c43cf3"
And it is installed system-wide, eg.:
sudo bundle install
sudo bundle show dolphin
/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/bundler/gems/dolphin-60e2e753edd6
When run 'bundle check' as App user (running puma / passenger, etc.), not as root, it will fail to recognize the system-wise installed gem:
bundle check
git://github.com/nengxu/dolphin.git (at 60e2e75) is not checked out. Please run bundle install
Previously general solution to this bug is to run 'bundle install --deployment'. However, this approach is just to avoid this problem, not to solve it.
I've done debugging on this annoying bug. And found it is due to the path setting of git-based gem:
/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/gems/bundler-1.3.5/lib/bundler/source/path.rb:112:in load_spec_files' expanded_path = path.expand_path(Bundler.root) raise expanded_path.inspect will give: /opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/gems/bundler-1.3.5/lib/bundler/source/path.rb:113:inload_spec_files': #Pathname:/home/neng/.bundler/ruby/2.0.0/dolphin-60e2e753edd6 (RuntimeError)
Obviously, Bundle only looks for the per-user bundle folder, not the system-wide folder of "/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/bundler/gems"
No wonder later:
/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/gems/bundler-1.3.5/lib/bundler/source/path.rb:143:in load_spec_files' raise PathError, "The path#{expanded_path}does not exist." which will caught by git.rb in line 176, and throw that exception we saw above: def load_spec_files super rescue PathError, GitError raise GitError, "#{to_s} is not checked out. Please runbundle install`"
end

Please correct this bug by searching also in system-wide bundler folder for git-based gems. Thanks.

Reply to this email directly or view it on GitHub:
#2492

[nengxu]

Bundler is one of the most useful gems. Thank you guys for creating and maintaining this wonderful tool. What I reported and commented are only meant to help you guys to make this tool even more useful.

We prefer to install gems system-wide so gems can be shared across projects. Bundler is supposed to help us to pick up the right gems from the pool of all installed gems as per each project's requirement (as in Gemfile). That's the reason why we use Bundler, right?

$GEM_HOME is usually set to per-user dir like: /home/neng/.gem/ruby/2.0.0
Accordingly, running "bundle install" or "bundle install --system", we will get the result:
bundle show dolphin
/home/neng/.gem/ruby/2.0.0/bundler/gems/dolphin-60e2e753edd6

So the option of "bundle install --system" is not working as expected too. We assume that "--system" will not install gems to a per-user dir, but to the system-wise dir, right?

Could you please tell me the right way in Bundler to install gems system-wise? Running "bundle install" as root will do, right? So "sudo bundle install" is also ok.

The problem is not like you said "sudo bundle install is not supported". The exception of not finding :git based gems from system-wise location is not consistent with the way Bundler is treating other gems which are not :git based but installed system-wide. For example, one dependency of the dolphin gem is parallel:
bundle show parallel
/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0/gems/parallel-0.6.5

So Bundler can load parallel from the system-wide dir. But it cannot do the same thing for :git based dolphin.

Please note, "bundle install --deployment" is not acceptable for us either because it will contradict the purpose of sharing gems across projects.

I'm using chruby, not rvm any more. In chruby, $GEM_PATH will give us both per-user location and system location:
echo $GEM_PATH
/home/neng/.gem/ruby/2.0.0:/opt/rubies/ruby-2.0.0-p195/lib/ruby/gems/2.0.0

Please when treating :git based gems, search through $GEM_PATH, not just looking in the per-user location. This approach, possibly the same you used with the not-git-based gems, will solve this long time bug. Thanks.

[nengxu]

Please fix this old bug. It has caused pain as shown here:
https://github.com/mongoid/mongoid/issues/3128

[xaviershay]

You said:

The problem is not like you said "sudo bundle install is not supported". The exception of not finding :git based gems from system-wise location is not consistent with the way Bundler is treating other gems

@indirect said

but that already happens if you don't use sudo to run bundle.

Sorry, we don't, and probably won't, support using sudo. Encouraging users to install gems as root is particularly dangerous, since any gem can run arbitrary code. If you really want to install system wide, set up your directory permissions / groups so that multiple users can write to it.

[nengxu]

@xaviershay, disappointed at your comment and decision. To repeat:
The problem is not like you said "sudo bundle install is not supported". The exception of not finding :git based gems from system-wise location is not consistent with the way Bundler is treating other gems

It is the user of Bundler (I am one of them) who should decide if he wants to use sudo or not. We understand the risk of using sudo.

If as you said, Bundler should discourage the users to use sudo, Bundler should not support "sudo bundle install" a not-git-base gem.

Why do you avoid to discuss the inconsistent behaviors between treating git-based and non-git-based gems? Thanks.

[dayglojesus]

I'd have to disagree with this @xaviershay -- it's not consistent.

If I have a gem source on Github and I run sudo bundle install, the gem gets installed!

But, when I run bundle check, it does not recognize that?

Bundler is installing the gem and ignoring that it's installed it. That's insane.

If Bundler isn't going to support sudo bundle operations for Github sources, then it sure as heck shouldn't be installing the designated gems in the first place.

[indirect]

Bundler doesn’t support sudo bundle operations period, if you read the documentation.

On Feb 21, 2014, at 8:35 AM, Brian Warsing notifications@github.com wrote:

I'd have to agree with this @xaviershay -- it's not consistent.

If I have a gem source on Github and I run sudo bundle install, the gem gets installed!

But, when I run bundle check, it does not recognize that?

Bundler is installing the gem and ignoring that it's installed it. That's insane.

If Bundler isn't going to support sudo bundle operations for Github sources, then it sure as heck shouldn't be installing the designated gems in the first place.

—
Reply to this email directly or view it on GitHub.

[nengxu]

Shall I repeat?

The problem is not like you said "sudo bundle install is not supported". The exception of not finding :git based gems from system-wise location is not consistent with the way Bundler is treating other gems (from rubygems)

Why do you avoid to discuss the inconsistent behaviors between treating git-based and non-git-based gems? Thanks.

[indirect]

Because it's consistent if you don't run "sudo bundle", and just run "bundle". Then Bundler is able to escalate it's own privileges if needed for some operations. The only way to create the problem you are describing is via "sudo bundle", which is not supported.

On Fri, Feb 21, 2014 at 11:58 AM, nengxu notifications@github.com wrote:

Shall I repeat?
The problem is not like you said "sudo bundle install is not supported". The exception of not finding :git based gems from system-wise location is not consistent with the way Bundler is treating other gems (from rubygems)

Why do you avoid to discuss the inconsistent behaviors between treating git-based and non-git-based gems? Thanks.

Reply to this email directly or view it on GitHub:
#2492 (comment)

[nengxu]

Please forget about "sudo bundle". Please verify this: inconsistent behaviors between treating git-based and non-git-based gems.

Apparently, with non-git-based gems, bundler will search through system-wise location. But for git-based gems, bundler did not search the system-wise location.

Thanks.

[indirect]

Git gems and non-git gems are of course not identical, but git gems are stored and used system-wide—unless you run "sudo bundle". Then they are not. So stop doing that.

On Sat, Feb 22, 2014 at 3:40 AM, nengxu notifications@github.com wrote:

Please forget about "sudo bundle". Please verify this: inconsistent behaviors between treating git-based and non-git-based gems.
Apparently, with non-git-based gems, bundler will search through system-wise location. But for git-based gems, bundler did not search the system-wise location.

Thanks.

Reply to this email directly or view it on GitHub:
#2492 (comment)

[nengxu]

What you said, "git gems are stored and used system-wide", is probably not correct. By default, 'bundle install' will put git based gems in the current user's ~/.gem.

/home/neng/.gem/jruby/1.9.3/bundler/gems/rails-f9e4c3c7c0c4

How can these gems which are under one user's home dir be used system-wide?

By "sudo bundle install", git based gems are installed in system-wide location:

/opt/rubies/jruby-1.7.9/lib/ruby/gems/shared/bundler/gems/rails-f9e4c3c7c0c4

So they can be used system-wide. That's exactly one reason why people want to "sudo bundle install".

Unfortunately, 'bundle check' will ignore the system-wide location when checking git based gems. And @indirect and @xaviershay are keeping arguing about "stop doing sudo bundle install". What a pity.

Please forget about "sudo bundle install", and focus on why "bundle check" did not look into system-wide locations for git-based gems. Thanks.

[indirect]

git gems (currently, at least) require write access at runtime. it's not possible to use root-owned files unless you run your web server as root. please don't do that.

On Sat, Feb 22, 2014 at 11:34 AM, nengxu notifications@github.com wrote:

What you said, "git gems are stored and used system-wide", is probably not correct. By default, 'bundle install' will put git based gems in the current user's ~/.gem.
/home/neng/.gem/jruby/1.9.3/bundler/gems/rails-f9e4c3c7c0c4
How can these gems which are under one user's home dir be used system-wide?
By "sudo bundle install", git based gems are installed in system-wide location:
/opt/rubies/jruby-1.7.9/lib/ruby/gems/shared/bundler/gems/rails-f9e4c3c7c0c4
So they can be used system-wide. That's exactly one reason why people want to "sudo bundle install".
Unfortunately, 'bundle check' will ignore the system-wide location when checking git based gems. And @indirect and @xaviershay are keeping arguing about "stop doing sudo bundle install". What a pity.

Please forget about "sudo bundle install", and focus on why "bundle check" did not look into system-wide locations for git-based gems. Thanks.

Reply to this email directly or view it on GitHub:
#2492 (comment)

[backwardm]

@nengxu: Did you ever find a solution to the problem you are describing? I'm currently debugging a similar situation where a github sourced gem (activeadmin) is not being recognized by Passenger. As you mentioned, the gem is being installed to [...]/bundler/gems/ rather than where the other gems are being installed. (I'm installing these into a user-level directory (using rbenv) using Bundler v1.7.2—no sudoing going on at all) :)

I've tried appending the [...]/bundler/gems path to my GEM_PATH, but that still doesn't help Passenger to see the github-sourced gem.

[indirect]

Don’t append […]/bundler/gems/ to your GEM_PATH, append […] to your GEM_PATH. In addition, Passenger (which is probably running as the nobody user) must have read and write permissions for git gem files.

On Aug 28, 2014, at 12:15 PM, Michael Davie notifications@github.com wrote:

@nengxu: Did you ever find a solution to the problem you are describing? I'm currently debugging a similar situation where a github sourced gem (activeadmin) is not being recognized by Passenger. As you mentioned, the gem is being installed to [...]/bundler/gems/ rather than where the other gems are being installed. (I'm installing these into a user-level directory (using rbenv) using Bundler v1.7.2—no sudoing going on at all) :)

I've tried appending the [...]/bundler/gems path to my GEM_PATH, but that still doesn't help Passenger to see the github-sourced gem.

—
Reply to this email directly or view it on GitHub.

[backwardm]

@indirect: Thanks for your note—and for the Bundler gem.

Passenger is running as my deploy user, so permissions (thankfully) weren't an issue. For now, I've stumbled on a way to get Passenger to recognize github-sourced gems: by adding the GEM_HOME environment variable to my site config file just below where I set the GEM_PATH variable.

Where I had this originally:

SetEnv GEM_PATH /home/deploy/.rbenv/versions/1.9.3-p194/gemsets/values

I added this:

SetEnv GEM_HOME /home/deploy/.rbenv/versions/1.9.3-p194/gemsets/values

Bundler wasn't at fault—Passenger just wasn't seeing the gems installed in the different location. Adding the GEM_HOME variable fixes that for some reason.

Now I need to learn why there is two seemingly-the-same environment variables: GEM_HOME & GEM_PATH & let the Passenger fellas know to update the end-of-install message to maybe include a note about that.

Going back to @nengxu's remarks—I wonder why Bundler installs github-sourced gems in a different location ([…]/bundler/gems/) than normally sourced gems which go directly into the /gems directory. I'll have to read the docs to learn about that.

[backwardm]

A quick note about this... I was able to determine that Bundler was not the problem in my case—even version 1.7.2 (latest as of today) installs github-sourced gems in the [...]/bundler/gems folder away from where it places 'normal' gems. Good to know.

I had a little bit of time to work on my particular issue further—to make a long story short, I was able to get Passenger to see the github-sourced gems by adding a GEM_HOME variable to my apache site config file. Once that additional environment variable is set, new github-sourced gems install and are recognized normally. No more vendor'd gems! Yay!

(To figure this out, I cloned a production server and worked on the clone. I broke it a few times and just refreshed the server using the saved image.)

The starting point was with all my gems vendor'd in: /vendor/cache, so first step was to undo that. The following were all done from the root of the Rails app which in my case is: /var/www/values.

• bundle config --delete disable_shared_gems
• bundle config --delete path
• rm -rf .bundle
• rm -rf vendor/cache

Now, add the GEM_HOME environment variable (the siteconfig file is named "values"):

• sudo nano /etc/apache2/sites-available/values

My original site config had this line in it per the Passenger end-of-install message:

• SetEnv GEM_PATH /home/deploy/.rbenv/versions/1.9.3-p194/gemsets/values

I added this line just after that one:

• SetEnv GEM_HOME /home/deploy/.rbenv/versions/1.9.3-p194/gemsets/values

Restarted apache:

• sudo service apache2 restart

Then bundled up again:

• bundle

Restarted Passenger:

• touch tmp/restart.txt

Now, I can specify any github-sourced gem and it works as expected. For example, I can now use a specific fork of a gem called papercrop (paperclip + jcrop):

• Change gem "papercrop" to gem "papercrop", github: 'ezuhaib/papercrop'
• bundle

Bundler will still install the gem in the odd location ([...]/bundler/gems/), but Passenger sees it just fine.

I still don't know why the extra GEM_HOME variable works—I need to read more of the Passenger docs.

Quick note about my setup:

Rails 4.0.4 app running on Ubuntu 12.04
ruby -v 1.9.3p194.
rbenv 0.4.0-98-g13a474c
apache 2.2.22
passenger 4.0.44 (passenger runs as my deploy:deploy user:group)
gems were installed via bundler 1.6.2 (now upgraded to 1.7.2)

[nengxu]

@backwardm I currently don't use any github gem in my projects. Have you tried other servers like Puma, Thin, etc? Will GEM_HOME be useful there? Thank you for sharing.

[backwardm]

@nengxu No, I am only using Passenger, so I don't know about Puma / Thin / Unicorn, etc. I suspect that the other application servers mentioned wouldn't necessarily have the same issue though. For some reason, adding that environment variable gave Passenger what it needed to "see" those gems. The other application servers might already know to look in the vicinity for github-sourced gems.

I understand what the difference is between the two variables are now (StackExchange), but haven't yet read up about Passenger's use / requirement of them.