Shareable Mini-Gemfiles?

[pboling]

I'm trying to use a layered config for my Gemfiles for CI (e.g. on Github Actions, Bitbucket Pipelines, CircleCI etc).

I hoped I could have a set of shared mini-gemfiles that I would use discretely for various CI parts, and which could be re-used across applications.

The mini-gemfiles would be for discrete contexts like "coverage", "debugging", "testing", "documentation", "style", etc. If this was to work, I'd create a Bundler plugin that would ship these shareable gemfiles, so integration would be a one liner (plugins auto-install, and auto-configure on a bundler hook).

But it turns out that Bundler's definition of eval_gemfile uses instance_eval, which changes the "current class" (or "default definee" if you like), and results in the self-same eval_gemfile being unavailable to the mini-Gemfile context. If it was to use class_eval instead the receiver would remain as the current class, and I think it would work. Also, I may be waaaay off base here, as to why it isn't working.

I'm wondering if this was an intentional decision, or just a chance, and also if there is a security consideration that would make a change to class_eval unwise.

Example

In the context of code coverage, we have to both run the test suite on a supported platform, and analyze the results.

We could have two mini-Gemfiles as follows, each providing in context documentation for various choices that may require deep knowledge of the gem library ecosystem, and which I don't want to have to find repeatedly for every project I create:

gemfiles/contexts/coverage.gemfile

# CodeCov + GitHub setup: https://github.com/marketplace/actions/codecov
# NOTE: CodeCov no longer supports integration via ruby gem!
#   see https://docs.codecov.com/docs/deprecated-uploader-migration-guide#ruby-uploader
# gem "codecov", "~> 0.6", require: false # For CodeCov
gem "simplecov", "~> 0.22", require: false
gem "simplecov-cobertura", require: false # XML for Jenkins
gem "simplecov-json", "~> 0.2", require: false # For CodeClimate
gem "simplecov-lcov", "~> 0.8", require: false
# SimpleCov extension gems can be slow to update.
# Avoid deprecations by using fixed forks.
gem "simplecov-rcov", github: "pboling/simplecov-rcov", branch: "patch-1", require: false

gemfiles/contexts/testing.gemfile

# Testing
gem "rspec", "~> 3.12"
gem "rspec-block_is_expected", "~> 1.0" # Ships a RuboCop rule
gem "rspec-stubbed_env", "~> 1.0"

gemfiles/coverage.gemfile

source "https://rubygems.org"

eval_gemfile "./contexts/core.gemfile"
eval_gemfile "./contexts/coverage.gemfile"
eval_gemfile "./contexts/testing.gemfile"

gemspec path: "../"

Why?

The basis for this idea is that I end up doing this in every project, and it is a significant amount of re-work, most of which I could standardize across all my projects.

https://gitlab.com/rubocop-lts/standard-rubocop-lts/-/tree/main/gemfiles

[segiddins]

I'm a bit unclear on what the issue is w/r/t eval_gemfile, and why that doesn't work for your needs? This also feels close to what https://github.com/segiddins/bundler-compose/tree/main is trying to solve (though not exactly the same interface)

[pboling]

I'm a bit unclear on what the issue is w/r/t eval_gemfile

To put it very simply, I am unable to call eval_gemfile from within a Gemfile that has itself been loaded via eval_gemfile. It appears that eval_gemfile is unavailable within its instance_exec.

why that doesn't work for your needs

Fundamentally I want composable gemfiles... and bundler-compose is awesome! It lacks a way to ship them to be shareable and then easily usable. Calling in to gemfile files stored within an installed gem in the GEM_HOME would be awful. Also I'm not sure if it can compose layers of gemfiles stacked one atop the other more than n=1 deep.

My example only went one deep, but it is oversimplified.

[pboling]

Just thought of another way to describe the idea.

I could make a gem that would ship a set of dependencies that relate to a specific context, much like the pry-suite gem is a set of pry-related gems, that you can easily grab as a set that work together as a souped-up IRB replacement.

This idea is similar, in two distinct parts.

First, make a set of individual gems, using an example namespace of "tan":

• tan-dev - would be a set of gems that I always want to include in the dev group of my Ruby projects' Gemfiles.
• tan-test - would be a set of gems that I always want to include in the test group of my Ruby projects' Gemfiles.

And then make another set of gems, using a sub-namespace of the first, "tan-dem":

• tan-dem-coverage - provides contextual group of deps to be loaded in a the CI pipeline that runs test coverage.
• tan-dem-testing - provides contextual group of deps to be loaded in a the CI pipeline that runs tests (with and without coverage - and not including any coverage gems).
• tan-dem-docs - provides contextual group of deps to be loaded in a the CI pipeline that runs documentation.
• tan-dem-style - provides contextual group of deps to be loaded in a the CI pipeline that runs style checks / linting.
• etc for several more contexts...

That is a lot of gems to build, and maintain... and each one is very thin, mostly existing to provide a set of dependencies that live in a gemspec... 😩

But what if...

That could all be replaced with a single gem that ships a set of re-usable gemfiles! 🤩

It won't work if gemfiles can't be eval'd multiple layers deep.

They can't use eval_gemfile themselves, as projects using them would already be loading the shareable gemfiles via eval_gemfile... 😢

[pboling]

Also of note: another downside of gemspecs as a "solution" for this use case is the lack of control over source (for good reason - I'm not complaining about that, nor do I think it should change - I've read the relevant threads).

My shareable gemfile gem concept is intended to encode a high level of understanding about specific context's library ecosystem. For example, simplecov-rcov emits deprecation warnings, and may never be fixed, thus we are relying on forks to fix problems. This can only be done in a Gemfile unless I hard fork the gem and release a fixed alternate. 😬

gem "simplecov-rcov", github: "pboling/simplecov-rcov", branch: "patch-1", require: false

In other words, accomplishing the goal via releasing a bunch of new gems with gemspecs, would force me to still copy paste lines like the one above into each projects Gemfile, which is exactly what I want to avoid (I maintain many dozens of public gems, and many-more-still private ones).

[pboling]

Gemfiles eval_gemfile'd from other gemfiles, eval_gemfile'd from still other gemfiles works perfectly in a Rails app where all the gemfiles are local to each other.

I guess I need to figure out why it isn't working in a stack of gems. The issue I had was definitely that eval_gemfile is not defined as a method though... so I'm a little confused.

[martinemde]

Ok, I'm imagining something like if bundler/inline could be composed. I thinking of inline because it "frees" bundler from the Gemfile/Gemfile.lock file structure and allows it to be inside of any ruby file. Now if you could require a file from a gem and it contained a bundler/inline, it would become part of the deps for the project. This isn't a perfect analogy, because I think you want to approach dependency installation and runtime activation and usage as different steps. Maybe you could "improve" the analogy if you gave this new file a .gemfile extension and only executed the steps when in the bundler parsing context?

Another way to express this might be: gemspecs don't support everything that a Gemfile does, and gems can't ship with more than one gemspec. Wouldn't it be nice if a gem could contain multiple Gemfile-like lists of dependencies accessible by name? I think if you could do that, the rails gem might be one of these (I'd imagine having sets of dependencies for different configurations of rails, like rspec, minitest, api only, etc). Right now this is composed for you using the rails new command.

I had read your idea before and it sounded very complex, but the I think your point about simplecov makes sense and made me rethink the idea. It could be nice to configure packages of dependencies like this. I guess the simplest version of this would be a directive in a Gemfile that loads another gemfile (from a gem or where?). Having to install gems before we can finish resolving seems odd though. How would you manage these sub-gemfiles?

[pboling]

Yes, a directive similar to eval_gemfile, but that can take an argument specifying the gem to load it from. I think given the existing nature of Bundler it would actually need to load it from a Bundler plugin gem.

Bundler plugins seem to already have access to a method like this:

Bundler::Plugin.gemfile_install("#{__dir__}/tan/dem/#{context}.gemfile")

I'm testing it out now, but it doesn't do what you would expect it to... at all. :/

Calling from within the relevant hook doesn't change the behavior.

      Bundler::Plugin.add_hook('before-install-all') do |dependencies|
        contexts.each do |context|
          Bundler::Plugin.gemfile_install("#{__dir__}/tan/dem/#{context}.gemfile")
        end
      end

Essentially it does nothing.

[pboling]

I also tried using gem_dir = Gem::Specification.find_by_name("tan-dem").gem_dir to find my plugin, in order to use the gem_dir with something like eval_gemfile("#{gem_dir}/tan/dem/coverage.gemfile"), but this seems to require that the plugin gem be installed prior, which entirely defeats the purpose of using a plugin.

[pboling]

This actually runs without errors, but once again, has no effect on the final Gemfile.lock (slightly pseudo):

      Bundler::Plugin.add_hook("before-install-all") do |dependencies|
        contexts.each do |context|
          dsl = Bundler::Dsl.new
          dsl.eval_gemfile("#{__dir__}/tan/dem/#{context}.gemfile")
        end
      end

[pboling]

An example where I've implemented the non-shared-pattern I am trying to replace with this shared-pattern is here, where I have:

• a set of top-level *.gemfile files which don't load the main Gemfile at all, but they do load the gemspec: https://github.com/rubocop-lts/standard-rubocop-lts/tree/main/gemfiles
• a set of a-la-carte contexts: https://github.com/rubocop-lts/standard-rubocop-lts/tree/main/gemfiles/contexts
• a set of workflows that use the top-level *.gemfile files explicitly: https://github.com/rubocop-lts/standard-rubocop-lts/tree/main/.github/workflows

I've already replicated this pattern many times, improving it a little each time, and it is getting hard to keep track of where the latest, best, version of the pattern is to copy to a new gem. Every time I find myself in this situation I know it is time to write a new gem to encode the pattern.

[pboling]

I almost have it working!

      Bundler::Plugin.add_hook("before-install-all") do |dependencies|
        active_contexts = ENV.fetch("GEMFILE_CONTEXTS", "").split(",")
        dsl = Bundler::Dsl.new
        active_contexts.each do |context|
          dsl.eval_gemfile("#{__dir__}/soup/gemfiles/#{context}.gemfile")
        end
        dependencies.concat(dsl.dependencies)
      end

Working

• Gemfile.lock is updated properly

Not Working

• Unable to use alternate sources for gems, such as git/github. When attempting bundle install it complains immediately that the gem source hasn't been checked out yet, which is really strange since bundle install is the time when it should do exactly that. I've come to terms with this limitation, so not a blocker.
• The contextually installed gems, even though they are installed, and are in the Gemfile.lock perfectly normally, are not considered part of the bundle when running commands other than install or update.
  For the following examples I have rake being installed from a "shareable mini-gemfile" via a code block like the above "before-install-all" hook.

Example 1

$ bundle exec rake
bundler: failed to load command: rake (/Users/pboling/.asdf/installs/ruby/3.2.2/bin/rake)
/Users/pboling/.asdf/installs/ruby/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/rubygems_integration.rb:308:in `block in replace_bin_path': can't find executable rake for gem rake. rake is not currently included in the bundle, perhaps you meant to add it to your Gemfile? (Gem::Exception)

Example 2

$ bundle info rake
Could not find gem 'rake'.

Example 3

I have constrained rake to version 12, so it should show as outdated re: version 13, but it does not.

$ bundle outdated
Resolving dependencies...
Fetching gem metadata from https://rubygems.org/..........

Bundle up to date!

Very Confused

What I don't understand is, why is being A) installed, and B) listed normally in the lockfile, not precisely, literally, and exclusively, the requirement for bundle CLI commands to work? What else do bundle CLI commands think they need?

[pboling]

How would you manage these sub-gemfiles?

@martinemde - I've pushed a POC that isn't working (except as noted in my last comment), but that is structured like what I'd hope to get working.

Now that it is pushed, and I switch the plugin to load from git, I get another very interesting error.

$ bundle install        
Fetching gem metadata from https://rubygems.org/
Fetching https://gitlab.com/kettle-rb/kettle-soup
Resolving dependencies...
Could not find compatible versions

Because every version of kettle-soup depends on version_gem >= 1.1.3, < 3
and version_gem >= 1.1.3, < 3 could not be found in https://gitlab.com/kettle-rb/kettle-soup
(at main@84d561b),
  kettle-soup cannot be used.
So, because Gemfile depends on kettle-soup >= 0,
  version solving has failed.

Can bundler plugins not have depeendencies? Or can they not be loaded from a git source?

[pboling]

I've also noticed that the before install all hook for a plugin does not run on a cold bundle install (i.e. first ever run), only on a warm run. This partially defeats the purpose of the hook (and entirely for my use case). :(