New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Top-level Gemfile in multi-gem project "leaks" dependencies #1668

Closed
dekellum opened this Issue Feb 19, 2012 · 6 comments

Comments

Projects
None yet
3 participants
@dekellum
Contributor

dekellum commented Feb 19, 2012

I maintain several multiple-gem in a single git-repo projects. Introducing Bundler in these projects was of great utility for:

  1. Better controlling per-gem dependencies, for example when running tests.
  2. Allowing cross-gem updates/testing without constantly needing to gem install incremental changes.

To enable the last in particular and keep things as easy to operate as possible, I'm using a single project top-level Gemfile. A much more prominent example of this style of Bundler usage is in rails:

https://github.com/rails/rails (see the top-level Gemfile)

There is a subtle flaw IMO in Bundler's behavior with these setups, in that any gem declared in any of the project gemspecs' is visible or require accessible from any other gem in the project, regardless of the gem's specific gemspec. This can lead to post-release surprises on gemspec declaration flaws, as Rubygems does not and could not have the same leaks.

The following sample project demonstrates this problem in tests:

https://github.com/dekellum/bundler-dep-leak

The alternative approach would be to use independent Gemfile's in each gem subdirectory. There are no dependency leaks in this case. See passing tests for the sample here:

https://github.com/dekellum/bundler-dep-leak/tree/sub-gemfiles

I find myself needing to switch to this per-Gemfile style setup periodically to test that my gemspec dependencies are correct and to fix any dependency issues. I then find myself switching back to take advantage of (2) above, and given that a single top-level Gemfile makes the project easier to use with a single top-level bundle install or bundle update. But this switching is proving tedious at best. It could be avoided by a Bundler fix for the dependency leaks, or restated as a feature request:

  • Bundler should restrain resolving/activating gems to only the regular and development dependencies declared directly by, or transitive regular dependencies of, the gemspec found/declared in the current working directory.
  • If no gemspec is found in the current working directory, the current behavior is preserved.
  • There is no change in the common case of a single Gemfile, single gemspec project.

At minimum I hope to shed light on this "leak" issue. If you guys agree that this is Bundler's issue or the above feature is desirable, I will be happy to attempt an implementation.

@indirect

This comment has been minimized.

Show comment
Hide comment
@indirect

indirect Feb 19, 2012

Member

Forgive me if I'm misunderstanding something here, but your example Gemfile explicitly declares that gems A, B, and C are all required. That means you have just told Bundler that all three of those gems, and every single child gem of all three of those gems, must be present and loadable once Bundler.setup has run. If that's not what you want, I think you need to write an explicit Gemfile for sub_b in the sub_b directory, and only use that Gemfile when you want to isolate your bundle to just sub_b and it's child deps.

Member

indirect commented Feb 19, 2012

Forgive me if I'm misunderstanding something here, but your example Gemfile explicitly declares that gems A, B, and C are all required. That means you have just told Bundler that all three of those gems, and every single child gem of all three of those gems, must be present and loadable once Bundler.setup has run. If that's not what you want, I think you need to write an explicit Gemfile for sub_b in the sub_b directory, and only use that Gemfile when you want to isolate your bundle to just sub_b and it's child deps.

@dekellum

This comment has been minimized.

Show comment
Hide comment
@dekellum

dekellum Feb 19, 2012

Contributor

Thanks for the quick response. I agree with your statement of fact. Here lies the gap between how bundler currently works and what I believe is desirable for a multi-gem project. See my sub-gemfiles sample branch (3rd link above). This is the only way to get completely correct dependency isolation. However by doing this, I lose:

  • The convenience of doing a single top level bundle install or update and will have complications in keeping the Gemfile.lock's in sync for patch releases.
  • The ability to make cross-gem changes and running the tests without constantly needing to gem install those changes.
Contributor

dekellum commented Feb 19, 2012

Thanks for the quick response. I agree with your statement of fact. Here lies the gap between how bundler currently works and what I believe is desirable for a multi-gem project. See my sub-gemfiles sample branch (3rd link above). This is the only way to get completely correct dependency isolation. However by doing this, I lose:

  • The convenience of doing a single top level bundle install or update and will have complications in keeping the Gemfile.lock's in sync for patch releases.
  • The ability to make cross-gem changes and running the tests without constantly needing to gem install those changes.
@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Feb 19, 2012

Contributor

Here's one alternate approach I've been using that allows you, I think, to "have your cake and eat it too".

The top-level Gemfile for the project is like this:

source 'https://rubygems.org'

group :development do
  PROJECT_ROOT = File.expand_path('..', __FILE__)

  # install all gems needed by sub-libs
  %w[ lib_a lib_b ].each do |lib|
    eval File.read(File.join(PROJECT_ROOT, lib, "Gemfile"))
  end
end

In effect, the top-level Gemfile simply reads in the Gemfiles of each subproject. Each subproject has its own gemfile. This ensures that when you are in a subproject, the Gemfile being used is the isolated one their with no extra dependencies. At the top level, you can use "bundler install" to install everything.

Contributor

myronmarston commented Feb 19, 2012

Here's one alternate approach I've been using that allows you, I think, to "have your cake and eat it too".

The top-level Gemfile for the project is like this:

source 'https://rubygems.org'

group :development do
  PROJECT_ROOT = File.expand_path('..', __FILE__)

  # install all gems needed by sub-libs
  %w[ lib_a lib_b ].each do |lib|
    eval File.read(File.join(PROJECT_ROOT, lib, "Gemfile"))
  end
end

In effect, the top-level Gemfile simply reads in the Gemfiles of each subproject. Each subproject has its own gemfile. This ensures that when you are in a subproject, the Gemfile being used is the isolated one their with no extra dependencies. At the top level, you can use "bundler install" to install everything.

@indirect

This comment has been minimized.

Show comment
Hide comment
@indirect

indirect Feb 19, 2012

Member

That's correct. Dependency isolation in Bundler is limited to per-Gemfile dependency trees, by design.

Rails (to use your "high profile" example from above) does in fact use the project Gemfile in this way, because the Gemfile corresponds to the entire Rails project, and not to any individual gem. The entire idea and point of Bundler is to manage your LOAD_PATH so that you can require any gem either listed at the top level of your Gemfile or depended on by one of those gems.

It sounds like what you're asking for is to be able to explicitly list a gem in your Gemfile, yet still be unable to load that gem. If that's true, I'm afraid we are unlikely to be willing to implement that, since that behaviour would be considered a critical bug.

If what you are after is simply letting your test processes make their own decisions about what gets required, you can easily disable Bundler's auto-require facility by adding :require => nil to each of the lines in your Gemfile. Once you have done that, only explicit requires in your own code will load things, and you should be able to run your tests unsullied by other libraries' code.

Member

indirect commented Feb 19, 2012

That's correct. Dependency isolation in Bundler is limited to per-Gemfile dependency trees, by design.

Rails (to use your "high profile" example from above) does in fact use the project Gemfile in this way, because the Gemfile corresponds to the entire Rails project, and not to any individual gem. The entire idea and point of Bundler is to manage your LOAD_PATH so that you can require any gem either listed at the top level of your Gemfile or depended on by one of those gems.

It sounds like what you're asking for is to be able to explicitly list a gem in your Gemfile, yet still be unable to load that gem. If that's true, I'm afraid we are unlikely to be willing to implement that, since that behaviour would be considered a critical bug.

If what you are after is simply letting your test processes make their own decisions about what gets required, you can easily disable Bundler's auto-require facility by adding :require => nil to each of the lines in your Gemfile. Once you have done that, only explicit requires in your own code will load things, and you should be able to run your tests unsullied by other libraries' code.

@dekellum

This comment has been minimized.

Show comment
Hide comment
@dekellum

dekellum Feb 19, 2012

Contributor

Thanks @myronmarston, I will definitely see how this plays as a practical workaround. I suspect though, that it is incremental: I will still have both top and sub-level Gemfile.lock's to contend with.

@indirect : I hope you will humor my making another attempt to make this issue and its consequences more clear:

Using the sub-gemfiles setup, in sub_b/lib/sub_b.rb, if I require 'hashie' and forget to declare it as sub_b.gemspec dependency, it will (correctly) fail. This is a really nice feature of using Bundler.

By comparison, by using the sample master branch setup, with one top-level Gemfile: The same code will not fail during development. I will only catch the gemspec problem once I've released it, and a consumer gets a LoadError if hashie is not installed (for example, consumer doesn't also use sub_c, which has the declared hashie dependency).

So currently I'm forced to trade the full set of intended Bundler features and conveniences (described above) to get proper detection of gemspec dependency omissions. It would be better if these multi-gem projects could have all the features with one setup. If you think my above stated feature request is too much magic, or too much of a change without an explicit opt in, then how about if we could preserve the top level Gemfile but add something like the following in a Gemfile to each of the three gem sub-directories:

gemspec :path => '.', :name => 'sub_b', :restrains => '..'

Where the meaning of :restrains would be: use '../Gemfile(.lock)' to resolve/activate all dependencies that are actually listed in sub_b.gemspec, but no other dependencies listed at the top level. And no Gemfile.lock's would be created at the sub-level.

Alternatively this could be rolled into the single, top level Gemfile, i,e:

source :rubygems

gemspec :path => 'sub_a', :name => 'sub_a', :constrain => true
gemspec :path => 'sub_b', :name => 'sub_b', :constrain => true
gemspec :path => 'sub_c', :name => 'sub_c', :constrain => true

With the same effect as above. Example: When I'm running rake/bundler/tests from the sub_b directory, then constrain dependencies to only gems listed in sub_b.gemspec plus transitive dependencies. I'd get the correct LoadError for require 'hashie' this way.

Contributor

dekellum commented Feb 19, 2012

Thanks @myronmarston, I will definitely see how this plays as a practical workaround. I suspect though, that it is incremental: I will still have both top and sub-level Gemfile.lock's to contend with.

@indirect : I hope you will humor my making another attempt to make this issue and its consequences more clear:

Using the sub-gemfiles setup, in sub_b/lib/sub_b.rb, if I require 'hashie' and forget to declare it as sub_b.gemspec dependency, it will (correctly) fail. This is a really nice feature of using Bundler.

By comparison, by using the sample master branch setup, with one top-level Gemfile: The same code will not fail during development. I will only catch the gemspec problem once I've released it, and a consumer gets a LoadError if hashie is not installed (for example, consumer doesn't also use sub_c, which has the declared hashie dependency).

So currently I'm forced to trade the full set of intended Bundler features and conveniences (described above) to get proper detection of gemspec dependency omissions. It would be better if these multi-gem projects could have all the features with one setup. If you think my above stated feature request is too much magic, or too much of a change without an explicit opt in, then how about if we could preserve the top level Gemfile but add something like the following in a Gemfile to each of the three gem sub-directories:

gemspec :path => '.', :name => 'sub_b', :restrains => '..'

Where the meaning of :restrains would be: use '../Gemfile(.lock)' to resolve/activate all dependencies that are actually listed in sub_b.gemspec, but no other dependencies listed at the top level. And no Gemfile.lock's would be created at the sub-level.

Alternatively this could be rolled into the single, top level Gemfile, i,e:

source :rubygems

gemspec :path => 'sub_a', :name => 'sub_a', :constrain => true
gemspec :path => 'sub_b', :name => 'sub_b', :constrain => true
gemspec :path => 'sub_c', :name => 'sub_c', :constrain => true

With the same effect as above. Example: When I'm running rake/bundler/tests from the sub_b directory, then constrain dependencies to only gems listed in sub_b.gemspec plus transitive dependencies. I'd get the correct LoadError for require 'hashie' this way.

@indirect

This comment has been minimized.

Show comment
Hide comment
@indirect

indirect Feb 22, 2012

Member

Okay, I think I see what you mean. You'd like Bundler to check the current working directory, and give you a subset of the parent directory's Gemfile based on which directory you are working from, right? That is definitely too much magic for me, sorry. :(

As "best practices for library maintainers", we suggest that the Gemfile.lock be added to .gitignore. I would add to that and suggest that you delete your Gemfile.lock before you do any pre-release testing, to effectively check and see if a vanilla gem install of your gem will work when users try it. If you adopt that pattern, then it seems like you should have all the upsides you described by simply adding Gemfiles to each subdirectory. When you are in the parent directory, you get the parent Gemfile loaded, and when you are in the child directories, you get the Gemfile that only loads the subdirectory's single gemspec.

Hopefully the reasoning behind deleting your Gemfile.lock before any pre-release checks run makes sense, and I would love to know if you have any further thoughts on this issue. Thanks!

Member

indirect commented Feb 22, 2012

Okay, I think I see what you mean. You'd like Bundler to check the current working directory, and give you a subset of the parent directory's Gemfile based on which directory you are working from, right? That is definitely too much magic for me, sorry. :(

As "best practices for library maintainers", we suggest that the Gemfile.lock be added to .gitignore. I would add to that and suggest that you delete your Gemfile.lock before you do any pre-release testing, to effectively check and see if a vanilla gem install of your gem will work when users try it. If you adopt that pattern, then it seems like you should have all the upsides you described by simply adding Gemfiles to each subdirectory. When you are in the parent directory, you get the parent Gemfile loaded, and when you are in the child directories, you get the Gemfile that only loads the subdirectory's single gemspec.

Hopefully the reasoning behind deleting your Gemfile.lock before any pre-release checks run makes sense, and I would love to know if you have any further thoughts on this issue. Thanks!

@indirect indirect closed this Feb 22, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment