gem exec rails new prints "Unresolved or ambiguous specs" warning

[mattbrictson]

Describe the problem as clearly as you can

In certain circumstances, gem exec will print a warning like this:

WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      reline (>= 0)
      Available/installed versions of this gem:
      - 0.3.8
      - 0.3.2
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.

Did you try upgrading RubyGems?

Yes.

$ gem --version
3.4.19

Post steps to reproduce the problem

The simplest way I've found to reproduce the problem is by using gem exec rails new with a Rails template that uses ask, a method from Thor that indirectly uses reline.

The following gist simply contains 1 line:

ask "Are we having fun yet?"

https://gist.github.com/mattbrictson/8f19b250ceb65ddec1c641ce1dc667fd

Invoking rails new with that gist yields the following:

$ gem exec rails new repro --no-rc -m https://gist.githubusercontent.com/mattbrictson/8f19b250ceb65ddec1c641ce1dc667fd/raw/418230976eb584c78bcb7e798c556256a461d5f0/template.rb
      create  
      create  README.md
      create  Rakefile
      create  .ruby-version
      create  config.ru
      create  .gitignore
      create  .gitattributes
      create  Gemfile
...
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      reline (>= 0)
      Available/installed versions of this gem:
      - 0.3.8
      - 0.3.2
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.

I can also reproduce the problem in a clean Docker image by first explicitly installing reline:

$ docker run -it ruby:3.2.2 /bin/sh
# gem install reline
Fetching reline-0.3.8.gem
Successfully installed reline-0.3.8
1 gem installed

A new release of RubyGems is available: 3.4.10 → 3.4.19!
Run `gem update --system 3.4.19` to update your installation.

# gem exec rails new repro --no-rc -m https://gist.githubusercontent.com/mattbrictson/8f19b250ceb65ddec1c641ce1dc667fd/raw/418230976eb584c78bcb7e798c556256a461d5f0/template.rb
...
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      reline (>= 0)
      Available/installed versions of this gem:
      - 0.3.8
      - 0.3.2
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.

Which command did you run?

gem exec

What were you expecting to happen?

I was expecting no warnings.

What actually happened?

WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      reline (>= 0)
      Available/installed versions of this gem:
      - 0.3.8
      - 0.3.2
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.

Run gem env and paste the output below

macOS

RubyGems Environment:
  - RUBYGEMS VERSION: 3.4.19
  - RUBY VERSION: 3.2.2 (2023-03-30 patchlevel 53) [x86_64-darwin22]
  - INSTALLATION DIRECTORY: /Users/mbrictson/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0
  - USER INSTALLATION DIRECTORY: /Users/mbrictson/.gem/ruby/3.2.0
  - RUBY EXECUTABLE: /Users/mbrictson/.rbenv/versions/3.2.2/bin/ruby
  - GIT EXECUTABLE: /usr/local/bin/git
  - EXECUTABLE DIRECTORY: /Users/mbrictson/.rbenv/versions/3.2.2/bin
  - SPEC CACHE DIRECTORY: /Users/mbrictson/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /Users/mbrictson/.rbenv/versions/3.2.2/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-darwin-22
  - GEM PATHS:
     - /Users/mbrictson/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0
     - /Users/mbrictson/.gem/ruby/3.2.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
     - "gem" => "--no-document"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /Users/mbrictson/.rbenv/versions/3.2.2/bin
     - /usr/local/Cellar/rbenv/1.2.0/libexec
     - /Users/mbrictson/.bin
     - .git/safe/../../bin
     - /Users/mbrictson/.rbenv/shims
     - /Users/mbrictson/.rbenv/bin
     - /usr/local/Cellar/pyenv-virtualenv/1.2.1/shims
     - /Users/mbrictson/.pyenv/shims
     - /Users/mbrictson/.nodenv/shims
     - /usr/local/opt/grep/libexec/gnubin
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/local/bin
     - /System/Cryptexes/App/usr/bin
     - /usr/bin
     - /bin
     - /usr/sbin
     - /sbin
     - /Library/Apple/usr/bin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin

Docker

RubyGems Environment:
  - RUBYGEMS VERSION: 3.4.10
  - RUBY VERSION: 3.2.2 (2023-03-30 patchlevel 53) [x86_64-linux]
  - INSTALLATION DIRECTORY: /usr/local/bundle
  - USER INSTALLATION DIRECTORY: /root/.local/share/gem/ruby/3.2.0
  - RUBY EXECUTABLE: /usr/local/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /usr/local/bundle/bin
  - SPEC CACHE DIRECTORY: /root/.local/share/gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /usr/local/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-linux
  - GEM PATHS:
     - /usr/local/bundle
     - /root/.local/share/gem/ruby/3.2.0
     - /usr/local/lib/ruby/gems/3.2.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
     - "install" => "--no-document"
     - "update" => "--no-document"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/bundle/bin
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin

[segiddins]

So I think this might be happening because rails new is running bundle install, and interestingly the warning seems to be printed under the turbo:install stimulus:install step -- it's actually happening in the Gem.clear_paths call after the body of gem exec has run. This I think is particular to reline being a default gem, because thor is using reline without declaring a dependency on it. Actually, there's also a level of indirection going in via https://rubygems.org/gems/readline as well. readline is being required (and therefore activated), but reline is never activated, but because it's a declared runtime dep of readline, it gets added to the list of unresolved deps.

In summary, yes this gets exhibited via gem exec (and we should fix it! thank you for the detailed bug report), but it also happens with gem install reline && ruby -rreadline -e 'Gem::Specification.reset' in ruby:3.2.2.

[segiddins]

More specifically, /usr/local/lib/ruby/3.2.0/aarch64-linux/readline.so is being successfully required by readline.rb, so reline is never required/activated. So actually, I'm not entirely sure this is a bug -- it might be the intended behavior, because we are indeed calling Gem::Specification.reset (via Gem.clear_paths) and there is an unresolved dep at that time.

[mattbrictson]

gem install reline && ruby -rreadline -e 'Gem::Specification.reset'

Interesting! That's a much more concise repro. Should I close this issue and replace it with a simpler one that describes the underlying readline/reline problem?

[mattbrictson]

Hmm, actually this doesn't seem to be uniquely a readline/reline problem. Any gem that declares a runtime dependency but doesn't use it right away will trigger warnings on Gem::Specification.reset. Is that supposed to happen?

$ ruby -rsidekiq -e 'Gem::Specification.reset'
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      rack (>= 2.2.4)
      Available/installed versions of this gem:
      - 3.0.8
      - 3.0.7
      - 2.2.8
      - 2.2.7
      - 2.2.6.4
      - 2.2.5
      - 2.2.4
      concurrent-ruby (< 2)
      Available/installed versions of this gem:
      - 1.2.2
      - 1.2.0
      - 1.1.10
      - 1.0.5

[segiddins]

I think that actually is the expected behavior?

[mattbrictson]

Thanks. So if this bug affects my executable that is being run via gem exec, I can work around it by explicitly adding require "reline" to my executable. Does that sound right?

[mattbrictson]

Something like this:

# Work around an "unresolved or ambiguous specs" warning when CLI is run via `gem exec`
at_exit do
  require "reline" if defined?(Readline)
end

[segiddins]

Alternatively, you could use Gem.finish_resolve to force an eager resolution of all unresolved deps. Or you could ignore the warning, up to you.

[indirect]

Does this bug need to be fixed in Rails, in Thor, or in the bundled Readline? I feel like we should probably let them know they are triggering this warning and how to fix it.

[mattbrictson]

I still don't fully understand the nature of this bug, or why Bundler needs to warn about unresolved specs at the end of gem exec.

However, I did some more investigation and found that:

• The warnings go away if I remove require "readline" from Thor or replace require "readline" with require "reline". Obviously this is not a good fix, because it would remove native readline functionality from Thor.
• Adding reline as a runtime dependency to Thor doesn't seem to help. Actually, it does. See my later comment: gem exec rails new prints "Unresolved or ambiguous specs" warning #6914 (comment)
• The problem seems to be that the readline gem declares a runtime dependency on reline¹ but doesn't use it if the native readline-ext can be loaded².

Footnotes

1. https://github.com/ruby/readline/blob/996af225ad7f91fb29f6921025c28e0f763be015/readline.gemspec#L32 ↩

2. https://github.com/ruby/readline/blob/996af225ad7f91fb29f6921025c28e0f763be015/lib/readline.rb#L2 ↩

[mattbrictson]

So I think this might be happening because rails new is running bundle install, and interestingly the warning seems to be printed under the turbo:install stimulus:install step

@segiddins if I modify the rails new command to add these options:

--minimal --skip-bundle

Then Rails does not run bundle install, nor does it run turbo:install, etc. Yet still I am getting the warning:

Details

$ gem exec --conservative --verbose rails new repro --minimal --skip-bundle --no-rc -m https://gist.githubusercontent.com/mattbrictson/8f19b250ceb65ddec1c641ce1dc667fd/raw/418230976eb584c78bcb7e798c556256a461d5f0/template.rb
running gem exec with:
	     version: >= 0
	conservative: true
	        args: ["new", "repro", "--minimal", "--skip-bundle", "--no-rc", "-m", "https://gist.githubusercontent.com/mattbrictson/8f19b250ceb65ddec1c641ce1dc667fd/raw/418230976eb584c78bcb7e798c556256a461d5f0/template.rb"]
	  executable: rails
	    gem_name: rails

activated rails (7.0.7.2)
      create  
      create  README.md
      create  Rakefile
      create  .ruby-version
      create  config.ru
      create  .gitignore
      create  .gitattributes
      create  Gemfile
         run  git init from "."
Initialized empty Git repository in /Users/mbrictson/Code/scratch/repro/.git/
      create  app
      create  app/assets/config/manifest.js
      create  app/assets/stylesheets/application.css
      create  app/channels/application_cable/channel.rb
      create  app/channels/application_cable/connection.rb
      create  app/controllers/application_controller.rb
      create  app/helpers/application_helper.rb
      create  app/jobs/application_job.rb
      create  app/mailers/application_mailer.rb
      create  app/models/application_record.rb
      create  app/views/layouts/application.html.erb
      create  app/views/layouts/mailer.html.erb
      create  app/views/layouts/mailer.text.erb
      create  app/assets/images
      create  app/assets/images/.keep
      create  app/controllers/concerns/.keep
      create  app/models/concerns/.keep
      create  bin
      create  bin/rails
      create  bin/rake
      create  bin/setup
      create  config
      create  config/routes.rb
      create  config/application.rb
      create  config/environment.rb
      create  config/puma.rb
      create  config/environments
      create  config/environments/development.rb
      create  config/environments/production.rb
      create  config/environments/test.rb
      create  config/initializers
      create  config/initializers/assets.rb
      create  config/initializers/content_security_policy.rb
      create  config/initializers/cors.rb
      create  config/initializers/filter_parameter_logging.rb
      create  config/initializers/inflections.rb
      create  config/initializers/new_framework_defaults_7_0.rb
      create  config/initializers/permissions_policy.rb
      create  config/locales
      create  config/locales/en.yml
      create  config/master.key
      append  .gitignore
      create  config/boot.rb
      create  config/database.yml
      create  db
      create  db/seeds.rb
      create  lib
      create  lib/tasks
      create  lib/tasks/.keep
      create  lib/assets
      create  lib/assets/.keep
      create  log
      create  log/.keep
      create  public
      create  public/404.html
      create  public/422.html
      create  public/500.html
      create  public/apple-touch-icon-precomposed.png
      create  public/apple-touch-icon.png
      create  public/favicon.ico
      create  public/robots.txt
      create  tmp
      create  tmp/.keep
      create  tmp/pids
      create  tmp/pids/.keep
      create  tmp/cache
      create  tmp/cache/assets
      create  vendor
      create  vendor/.keep
      create  test/fixtures/files
      create  test/fixtures/files/.keep
      create  test/controllers
      create  test/controllers/.keep
      create  test/mailers
      create  test/mailers/.keep
      create  test/models
      create  test/models/.keep
      create  test/helpers
      create  test/helpers/.keep
      create  test/integration
      create  test/integration/.keep
      create  test/channels/application_cable/connection_test.rb
      create  test/test_helper.rb
      remove  app/jobs
      remove  app/views/layouts/mailer.html.erb
      remove  app/views/layouts/mailer.text.erb
      remove  app/mailers
      remove  test/mailers
      remove  app/javascript/channels
      remove  app/channels
      remove  test/channels
      remove  config/initializers/cors.rb
      remove  config/initializers/new_framework_defaults_7_0.rb
       apply  https://gist.githubusercontent.com/mattbrictson/8f19b250ceb65ddec1c641ce1dc667fd/raw/418230976eb584c78bcb7e798c556256a461d5f0/template.rb
  Are we having fun yet? yes
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      reline (>= 0)
      Available/installed versions of this gem:
      - 0.3.8
      - 0.3.7
      - 0.3.2
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.

[mattbrictson]

Adding reline as a runtime dependency to Thor doesn't seem to help.

Actually, I take this back. I can't figure out an easy way to test this change. My edits to the gemspec at ~/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/thor-1.2.2/thor.gemspec don't seem to have any effect.

[mattbrictson]

Ah, I didn't realize that the spec is loaded from a different path than the gem itself. It seems to be loaded from:

~/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/specifications/thor-1.2.2.gemspec

If I edit that gemspec and add readline as a runtime dependency, then the warning goes away.

So it seems like the solution may be for Thor to add readline as a dependency.