Bundler.with_clean_env and vendorized gems #1981

sethvargo · 2012-06-13T04:05:02Z

Overview

When shelling out with a bundler subshell, bundler cannot find vendorized gems.

Example

bundle install --path vendor/gems

class A
  def test
    Bundler.with_clean_env do
      system "bundle show bundler" # run any bundle commd here
    end
  end
end

This will throw an error saying dependency ... to_specs Could not find gem bundler amongst [...]

The text was updated successfully, but these errors were encountered:

indirect · 2012-06-13T04:06:33Z

Yes. That is the entire point of with_clean_env. If you want to use gems from a path, she'll out directly: system 'bundle show bundler'.

On Jun 12, 2012, at 9:05 PM, Seth Vargoreply@reply.github.com wrote:

Overview

When shelling out with a bundler subshell, bundler cannot find vendorized gems.

Example
bundle install --path vendor/gems
class A
 def test
   Bundler.with_clean_env do
     system "bundle show bundler" # run any bundle commd here
   end
 end
end
This will throw an error saying dependency ... to_specs Could not find gem bundler amongst [...]

Reply to this email directly or view it on GitHub:
#1981

sethvargo · 2012-06-13T04:19:40Z

Not if I'm trying to run something like bundle install... Thoughtbot's Appraisal gem is a good example. I can't just shell out, because Bundler will complain that the specs have already been loaded. I can't shell out with_clean_env because it can't find my vendorized gems. It's worth noting that it works with_clean_env when gems are installed globally.

indirect · 2012-06-13T04:26:53Z

What do you mean by "Bundler will complain that the specs have already been loaded"?

indirect · 2012-06-13T04:27:51Z

For what it's worth, it isn't currently supported to run bundle install from inside a Bundled environment, although I have heard of people doing it.

sethvargo · 2012-06-13T13:46:50Z

($) represents the shell command run inside the class above:

Dirty environment

$ bundle install --gemfile='/Users/svargo/Development/project/gemfiles/rails_3_0.gemfile'
Unfortunately, a fatal error has occurred. Please report this error to the Bundler issue tracker at https://github.com/carlhuda/bundler/issues so that we can fix it. Please include the full output of the command, your Gemfile and Gemfile.lock. Thanks!
/Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/definition.rb:119:in `resolve_remotely!': Specs already loaded (RuntimeError)
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/installer.rb:48:in `run'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/installer.rb:12:in `install'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/cli.rb:220:in `install'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/vendor/thor/task.rb:22:in `send'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/vendor/thor/task.rb:22:in `run'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/vendor/thor.rb:263:in `dispatch'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/vendor/thor/base.rb:386:in `start'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/bin/bundle:13
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/bin/bundle:23:in `load'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/bin/bundle:23

Clean environment

$ bundle install --gemfile='/Users/svargo/Development/project/gemfiles/rails_3_0.gemfile'
/Users/svargo/.rvm/rubies/ree-1.8.7-2012.02/lib/ruby/site_ruby/1.8/rubygems/dependency.rb:247:in `to_specs': Could not find bundler (>= 0) amongst [actionmailer-3.1.6, actionpack-3.1.6, activemodel-3.1.6, activerecord-3.1.6, activeresource-3.1.6, activesupport-3.1.6, arel-2.2.3, builder-3.0.0, chunky_png-1.2.5, compass-0.11.7, deep_merge-1.0.0, erubis-2.7.0, fssm-0.2.9, git-1.2.5, hike-1.2.1, httpclient-2.2.5, i18n-0.6.0, jeweler-1.6.4, json-1.7.3, mail-2.3.3, metaclass-0.0.1, mime-types-1.18, mocha-0.10.5, multi_json-1.2.0, mustache-0.99.4, navigation_builder-2.0.0.beta1, nokogiri-1.5.4, polyglot-0.3.3, pygmentize-0.0.3, rack-1.3.6, rack-cache-1.2, rack-mount-0.8.3, rack-ssl-1.3.2, rack-test-0.6.1, rails-3.1.6, railties-3.1.6, rake-0.9.2.2, rcov-0.9.11, rdoc-3.12, redcarpet-2.1.1, redgreen-1.2.2, rocco-0.8.2, sass-3.1.19, shoulda-3.0.1, shoulda-addons-0.2.3, shoulda-context-1.0.0, shoulda-matchers-1.0.0, smusher-0.4.9, sprockets-2.0.4, sqlite3-1.3.6, thor-0.14.6, tilt-1.3.3, treetop-1.4.10, tzinfo-0.3.33] (Gem::LoadError)
    from /Users/svargo/.rvm/rubies/ree-1.8.7-2012.02/lib/ruby/site_ruby/1.8/rubygems/dependency.rb:256:in `to_spec'
    from /Users/svargo/.rvm/rubies/ree-1.8.7-2012.02/lib/ruby/site_ruby/1.8/rubygems.rb:1231:in `gem'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/bin/bundle:22

What's worth noting also: if I shell out twice, in a "dirty" environment, the first will fail, but the second will succeed:

$ bundle install --gemfile='/Users/svargo/Development/project/gemfiles/rails_3_0.gemfile'
Unfortunately, a fatal error has occurred. Please report this error to the Bundler issue tracker at https://github.com/carlhuda/bundler/issues so that we can fix it. Please include the full output of the command, your Gemfile and Gemfile.lock. Thanks!
/Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/definition.rb:119:in `resolve_remotely!': Specs already loaded (RuntimeError)
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/installer.rb:48:in `run'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/installer.rb:12:in `install'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/cli.rb:220:in `install'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/vendor/thor/task.rb:22:in `send'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/vendor/thor/task.rb:22:in `run'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/vendor/thor.rb:263:in `dispatch'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/lib/bundler/vendor/thor/base.rb:386:in `start'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/gems/bundler-1.1.3/bin/bundle:13
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/bin/bundle:23:in `load'
    from /Users/svargo/.rvm/gems/ree-1.8.7-2012.02@global/bin/bundle:23


$ bundle install --gemfile='/Users/svargo/Development/project/gemfiles/rails_3_1.gemfile'
Using rake (0.9.2.2) 
(... suppressed ...)
Using sqlite3 (1.3.6) 
Your bundle is complete! It was installed into ./vendor/bundle

sethvargo · 2012-06-13T13:48:56Z

Very similar: #1569

indirect · 2012-06-13T16:45:11Z

@hone, any idea why the specs would already be loaded the first time a subshell bundler is invoked?

sethvargo · 2012-06-14T13:40:51Z

It looks like this used to work in 1.0.0

indirect · 2012-06-14T17:38:18Z

Well, fwiw, 1.0.0 has known bugs in with_clean_env that mean it doesn't clean the env. So that may be what you're seeing.

On Jun 14, 2012, at 6:40 AM, Seth Vargoreply@reply.github.com wrote:

It looks like this used to work in 1.0.0

Reply to this email directly or view it on GitHub:
#1981 (comment)

sethvargo · 2012-06-14T18:38:04Z

But the bug exists in 1.1, not 1.0... Seems counterintuitive to call it a "bug" if that's the case :(

indirect · 2012-06-14T19:19:14Z

You should be able to shell out from inside with_clean_env and have things work. There was recently a fix for GEM_PATH inside with_clean_env that hasn't been released yet. Can you try building Bundler from master, and see if that fixes your issue? If not, reproduction steps would be great so that we can fix the shelling out case. Thanks.

trinary · 2013-03-01T18:42:53Z

Is this still an issue? Do we have a simple way to reproduce it?

sethvargo · 2013-03-01T18:53:33Z

I no longer have the project this was occurring in. We moved off appraisal and no longer saw the issue.

sdrye · 2013-07-12T15:52:59Z

I've encountered something similar. I have two apps installed with "bundle install --deployment", and AppA needs to call a script in AppB. When AppA shells out inside Bundler.with_clean_env to call "bundle exec AppB/bin/script", I get the following error:

/opt/ruby-enterprise/lib/ruby/site_ruby/1.8/rubygems/dependency.rb:247:in `to_specs': Could not find bundler (>= 0) amongst [...] (Gem::LoadError)
        from /opt/ruby-enterprise/lib/ruby/site_ruby/1.8/rubygems/dependency.rb:256:in `to_spec'
        from /opt/ruby-enterprise/lib/ruby/site_ruby/1.8/rubygems.rb:1208:in `gem'
        from /opt/ruby-enterprise/bin/bundle:18

/opt/ruby-enterprise/bin/gem -v is 1.8.15.
Bundler version 1.3.5
A maybe-related issue is that using "export BUNDLE_GEMFILE=AppB/Gemfile && bundle exec AppB/bin/script" doesn't use the right Gemfile (it uses AppA's, not AppB's). I had to use a "Dir.chdir "AppB" do" inside the with_clean_env to get it to use AppB's Gemfile.

indirect · 2013-07-12T17:05:52Z

@sdrye, I suggest something like this:

Bundler.with_clean_env do
  %x{BUNDLE_GEMFILE=AppB/Gemfile bundle exec AppB/script/foo}
end

sdrye · 2013-07-12T21:51:19Z

Changing the command line to place the environment variables onto the bundle exec command line (instead of using the original "export BUNDLE_GEMFILE && bundle exec /opt/AppB/bin/script") also fails.

Bundler.with_clean_env do
%x{BUNDLE_GEMFILE=/opt/AppB/Gemfile bundle exec /opt/AppB/bin/script}
end

was what we had originally. That failed, it could not find the bundler gem and in addition it used the wrong Gemfile (this was clear from the "amongst" array, which contained the gems from AppA) .

Switching to

Bundler.with_clean_env do
  Dir.chdir /opt/AppB do
    %x{bundle exec /opt/AppB/bin/script}
  end
end

picked up the right gemfile (the contents of the "amongst" list matched the Gemfile for AppB this time), but again complains that "Could not find bundler (>= 0) amongst [...]", the exception trace that's in the report above. Using

%x{BUNDLE_GEMFILE=/opt/AppB/Gemfile bundle exec /opt/AppB/bin/script}

inside the Dir.chdir gets the same error.

On irc this morning I got the suggestion to try adding bash -c, that failed as well, in the same way.

indirect · 2013-07-12T22:02:35Z

That is extremely strange. The most likely option is that you're nesting shells in such a way that Bundler no longer has access to the original, un-bundled ENV variables. If you shell out to something that shells out to bundler, for example, that could happen.

sdrye · 2013-07-15T15:19:31Z

Our "AppA" is a straightforward Rails 2.3.18 app, so it shouldn't be doing anything strange (other than being 2.3.x). That said, I've written two little command line apps to try to reproduce the issue and can't. I'll keep trying to come up with a reproducible test case.

indirect · 2013-07-15T16:06:53Z

Yeah... That also makes me suspect that this is a double-shellout case. Is it possible that your production servers are getting launched by code that has a loaded bundle? If they inherit the env from a parent process, that might do it. with_clean_env will only be able to restore one level of env up from where it is run.

On Mon, Jul 15, 2013 at 8:21 AM, sdrye notifications@github.com wrote:

Our "AppA" is a straightforward Rails 2.3.18 app, so it shouldn't be doing anything strange (other than being 2.3.x). That said, I've written two little command line apps to try to reproduce the issue and can't. I'll keep trying to come up with a reproducible test case.

Reply to this email directly or view it on GitHub:
#1981 (comment)

sdrye · 2013-07-16T15:11:29Z

Have you heard of anyone having issues using bundler with thin? I've narrowed it down to the point where if I run "bundle exec script/server thin -e production" it works, but if I run "bundle exec thin start -d --config=/etc/thin/AppA" it fails.

sethvargo · 2013-07-16T15:17:59Z

@sdrye just jumpin in because this one looks familiar. There are known issues with bundler not passing all the config flags. You could try surrounding it with quotes:

$ bundle exec 'thin start -d --config=/...'

sdrye · 2013-07-16T15:33:10Z

No joy with the quotes, unfortunately. It appears to be an issue when thin creates multiple servers. So "bundle exec thin start -e production" works, but "bundle exec thin start -e production -s 4" doesn't.

sdrye · 2013-07-16T15:40:58Z

Doesn't seem to be related to how thin daemonizes the app, either. "bundle exec thin start -e production -p 3000 -d --pid tmp/pids/thin.3000.pid" plus "bundle exec thin start -e production -p 3001 -d --pid tmp/pids/thin.3001.pid" gives me two servers both of which work. So it's very specific to thin starting multiple servers by itself.

indirect · 2013-07-16T18:07:24Z

I don't have any experience with how thin starts multiple servers, but I haven't seen any previous reports of Bundler interfering with either forking or threading servers (like Puma, or Unicorn, Phusion Passenger). You could try executing thin without using the bundle command, running something like ruby -rbundler/setup -S thin start .... That would at least clarify whether it's an issue with exec.

sdrye · 2013-07-17T13:49:03Z

I've found the cause. It's in the way that thin's "cluster" mode starts servers. It's what you thought it might be, @indirect, a double-shellout. When you start thin with servers: or --servers, it goes through

https://github.com/macournoyer/thin/blob/v1.5.1/lib/thin/controllers/cluster.rb

Which then calls "run" in

https://github.com/macournoyer/thin/blob/v1.5.1/lib/thin/command.rb

In this second one you can see that it uses "shellify" to create a new command line for launching the child server and calls popen3 rather than just forking. The command line it creates is just a straight call to "thin", without bundle exec. So the outer, initial process is called with "bundle exec thin start", but then it starts the children with just "thin start". So it looks like with_clean_env can't work correctly with thin in cluster mode (at least at 1.5.1, not sure yet about 2.0.0-pre) because of how thin implemented cluster mode. Unfortunately cluster mode is what thin does when you set it up in a production config with a config file in /etc/thin. At least there's a workaround, though, where you can start each of the servers individually with --port --pid and --log.

indirect · 2013-07-17T19:44:08Z

Ahh, that makes sense. That also likely explains why I haven't seen this reported before, since everyone I've talked to who uses thin manages multiple processes without using thin's cluster mode. Thanks for tracking it down! If you really want to use thins's cluster mode, you can probably work around it by writing a script that does the same thing as thin, but uses with_clean_env around the open3.

On Wed, Jul 17, 2013 at 6:49 AM, sdrye notifications@github.com wrote:

I've found the cause. It's in the way that thin's "cluster" mode starts servers. It's what you thought it might be, @indirect, a double-shellout. When you start thin with servers: or --servers, it goes through
https://github.com/macournoyer/thin/blob/v1.5.1/lib/thin/controllers/cluster.rb
Which then calls "run" in
https://github.com/macournoyer/thin/blob/v1.5.1/lib/thin/command.rb

In this second one you can see that it uses "shellify" to create a new command line for launching the child server and calls popen3 rather than just forking. The command line it creates is just a straight call to "thin", without bundle exec. So the outer, initial process is called with "bundle exec thin start", but then it starts the children with just "thin start". So it looks like with_clean_env can't work correctly with thin (at least at 1.5.1, not sure yet about 2.0.0-pre) because of how thin implemented cluster mode. Unfortunately cluster mode is what thin does when you set it up in a production config with a config file in /etc/thin. At least there's a workaround, though, where you can start each of the servers individually with --port --pid and --log.

Reply to this email directly or view it on GitHub:
#1981 (comment)

indirect closed this as completed Jul 12, 2013

ejhayes mentioned this issue Feb 18, 2015

Must explicitly specify gemfile if using bundler inside hook scripts aws/aws-codedeploy-agent#7

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundler.with_clean_env and vendorized gems #1981

Bundler.with_clean_env and vendorized gems #1981

sethvargo commented Jun 13, 2012

indirect commented Jun 13, 2012

Overview

Example

sethvargo commented Jun 13, 2012

indirect commented Jun 13, 2012

indirect commented Jun 13, 2012

sethvargo commented Jun 13, 2012

sethvargo commented Jun 13, 2012

indirect commented Jun 13, 2012

sethvargo commented Jun 14, 2012

indirect commented Jun 14, 2012

sethvargo commented Jun 14, 2012

indirect commented Jun 14, 2012

trinary commented Mar 1, 2013

sethvargo commented Mar 1, 2013

sdrye commented Jul 12, 2013

indirect commented Jul 12, 2013

sdrye commented Jul 12, 2013

indirect commented Jul 12, 2013

sdrye commented Jul 15, 2013

indirect commented Jul 15, 2013

Our "AppA" is a straightforward Rails 2.3.18 app, so it shouldn't be doing anything strange (other than being 2.3.x). That said, I've written two little command line apps to try to reproduce the issue and can't. I'll keep trying to come up with a reproducible test case.

sdrye commented Jul 16, 2013

sethvargo commented Jul 16, 2013

sdrye commented Jul 16, 2013

sdrye commented Jul 16, 2013

indirect commented Jul 16, 2013

sdrye commented Jul 17, 2013

indirect commented Jul 17, 2013

Bundler.with_clean_env and vendorized gems #1981

Bundler.with_clean_env and vendorized gems #1981

Comments

sethvargo commented Jun 13, 2012

Overview

Example

indirect commented Jun 13, 2012

Overview

Example

sethvargo commented Jun 13, 2012

indirect commented Jun 13, 2012

indirect commented Jun 13, 2012

sethvargo commented Jun 13, 2012

Dirty environment

Clean environment

sethvargo commented Jun 13, 2012

indirect commented Jun 13, 2012

sethvargo commented Jun 14, 2012

indirect commented Jun 14, 2012

sethvargo commented Jun 14, 2012

indirect commented Jun 14, 2012

trinary commented Mar 1, 2013

sethvargo commented Mar 1, 2013

sdrye commented Jul 12, 2013

indirect commented Jul 12, 2013

sdrye commented Jul 12, 2013

indirect commented Jul 12, 2013

sdrye commented Jul 15, 2013

indirect commented Jul 15, 2013

Our "AppA" is a straightforward Rails 2.3.18 app, so it shouldn't be doing anything strange (other than being 2.3.x). That said, I've written two little command line apps to try to reproduce the issue and can't. I'll keep trying to come up with a reproducible test case.

sdrye commented Jul 16, 2013

sethvargo commented Jul 16, 2013

sdrye commented Jul 16, 2013

sdrye commented Jul 16, 2013

indirect commented Jul 16, 2013

sdrye commented Jul 17, 2013

indirect commented Jul 17, 2013