bundle update --conservative performs several un-necessary updates (Including MAJORS)

[luke-hill]

Describe the problem as clearly as you can

Running bundle update --conservative on my gemfile performs an un-necessary major version update to a dependent gem.

Post steps to reproduce the problem

Gemfile

# frozen_string_literal: true

source "https://rubygems.org"

ruby "2.7.2"

gem "puma"
gem "rails", "~> 6.0.3"

gem "haml-rails"
gem "liquid", "4.0.3"
gem "meta-tags", "~> 2.14"
gem "react-rails"
gem "rich_text_renderer"
gem "turbolinks"
gem "view_component", require: "view_component/engine"
gem "webpacker"

gem "contentful"
gem "redis"

group :development do
  # Access an interactive console on exception pages or by calling 'console' anywhere in the code.
  gem "haml_lint", require: false
  gem "listen"
  gem "web-console"
  # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
  gem "spring"
  gem "spring-watcher-listen"
end

group :test do
  gem "capybara", "~> 3.8"
  gem "climate_control"
  gem "cucumber", "~> 4.0"
  gem "cucumber-rails", "~> 2.0"
  gem "mailosaur", "~> 7.0"
  gem "retriable", "~> 3.1"
  gem "rspec"
  gem "rspec-rails"
  gem "selenium-webdriver", "4.0.0.alpha6"
  gem "simplecov", require: false
  gem "site_prism", "~> 3.7"
  gem "webdrivers", "~> 4.4"
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem "tzinfo-data", platforms: %i[mingw mswin x64_mingw jruby]

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    actioncable (6.0.3.4)
      actionpack (= 6.0.3.4)
      nio4r (~> 2.0)
      websocket-driver (>= 0.6.1)
    actionmailbox (6.0.3.4)
      actionpack (= 6.0.3.4)
      activejob (= 6.0.3.4)
      activerecord (= 6.0.3.4)
      activestorage (= 6.0.3.4)
      activesupport (= 6.0.3.4)
      mail (>= 2.7.1)
    actionmailer (6.0.3.4)
      actionpack (= 6.0.3.4)
      actionview (= 6.0.3.4)
      activejob (= 6.0.3.4)
      mail (~> 2.5, >= 2.5.4)
      rails-dom-testing (~> 2.0)
    actionpack (6.0.3.4)
      actionview (= 6.0.3.4)
      activesupport (= 6.0.3.4)
      rack (~> 2.0, >= 2.0.8)
      rack-test (>= 0.6.3)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.0, >= 1.2.0)
    actiontext (6.0.3.4)
      actionpack (= 6.0.3.4)
      activerecord (= 6.0.3.4)
      activestorage (= 6.0.3.4)
      activesupport (= 6.0.3.4)
      nokogiri (>= 1.8.5)
    actionview (6.0.3.4)
      activesupport (= 6.0.3.4)
      builder (~> 3.1)
      erubi (~> 1.4)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.1, >= 1.2.0)
    activejob (6.0.3.4)
      activesupport (= 6.0.3.4)
      globalid (>= 0.3.6)
    activemodel (6.0.3.4)
      activesupport (= 6.0.3.4)
    activerecord (6.0.3.4)
      activemodel (= 6.0.3.4)
      activesupport (= 6.0.3.4)
    activestorage (6.0.3.4)
      actionpack (= 6.0.3.4)
      activejob (= 6.0.3.4)
      activerecord (= 6.0.3.4)
      marcel (~> 0.3.1)
    activesupport (6.0.3.4)
      concurrent-ruby (~> 1.0, >= 1.0.2)
      i18n (>= 0.7, < 2)
      minitest (~> 5.1)
      tzinfo (~> 1.1)
      zeitwerk (~> 2.2, >= 2.2.2)
    addressable (2.7.0)
      public_suffix (>= 2.0.2, < 5.0)
    ast (2.4.1)
    babel-source (5.8.35)
    babel-transpiler (0.7.0)
      babel-source (>= 4.0, < 6)
      execjs (~> 2.0)
    bindex (0.8.1)
    builder (3.2.4)
    capybara (3.34.0)
      addressable
      mini_mime (>= 0.1.3)
      nokogiri (~> 1.8)
      rack (>= 1.6.0)
      rack-test (>= 0.6.3)
      regexp_parser (~> 1.5)
      xpath (~> 3.2)
    childprocess (3.0.0)
    climate_control (0.2.0)
    concurrent-ruby (1.1.7)
    connection_pool (2.2.3)
    contentful (2.15.4)
      http (> 0.8, < 5.0)
      multi_json (~> 1)
    crass (1.0.6)
    cucumber (4.1.0)
      builder (~> 3.2, >= 3.2.3)
      cucumber-core (~> 7.1, >= 7.1.0)
      cucumber-create-meta (~> 1.0.0, >= 1.0.0)
      cucumber-cucumber-expressions (~> 10.1, >= 10.1.0)
      cucumber-gherkin (~> 14.0, >= 14.0.1)
      cucumber-html-formatter (~> 7.0, >= 7.0.0)
      cucumber-messages (~> 12.2, >= 12.2.0)
      cucumber-wire (~> 3.1, >= 3.1.0)
      diff-lcs (~> 1.3, >= 1.3, < 1.4)
      multi_test (~> 0.1, >= 0.1.2)
      sys-uname (~> 1.0, >= 1.0.2)
    cucumber-core (7.1.0)
      cucumber-gherkin (~> 14.0, >= 14.0.1)
      cucumber-messages (~> 12.2, >= 12.2.0)
      cucumber-tag-expressions (~> 2.0, >= 2.0.4)
    cucumber-create-meta (1.0.0)
      cucumber-messages (~> 12.2, >= 12.2.0)
      sys-uname (~> 1.2, >= 1.2.1)
    cucumber-cucumber-expressions (10.3.0)
    cucumber-gherkin (14.2.0)
      cucumber-messages (~> 12.4, >= 12.4.0)
    cucumber-html-formatter (7.2.0)
      cucumber-messages (~> 12.4, >= 12.4.0)
    cucumber-messages (12.4.0)
      protobuf-cucumber (~> 3.10, >= 3.10.8)
    cucumber-rails (2.2.0)
      capybara (>= 2.18, < 4)
      cucumber (>= 3.0.2, < 6)
      mime-types (~> 3.2)
      nokogiri (~> 1.8)
      rails (>= 5.0, < 7)
    cucumber-tag-expressions (2.0.4)
    cucumber-wire (3.1.0)
      cucumber-core (~> 7.1, >= 7.1.0)
      cucumber-cucumber-expressions (~> 10.1, >= 10.1.0)
      cucumber-messages (~> 12.2, >= 12.2.0)
    diff-lcs (1.4.4)
    docile (1.3.5)
    domain_name (0.5.20190701)
      unf (>= 0.0.5, < 1.0.0)
    erubi (1.10.0)
    erubis (2.7.0)
    execjs (2.7.0)
    faraday (1.0.0)
      multipart-post (>= 1.2, < 3)
    ffi (1.14.2)
    ffi-compiler (1.0.1)
      ffi (>= 1.0.0)
      rake
    globalid (0.4.2)
      activesupport (>= 4.2.0)
    haml (5.2.1)
      temple (>= 0.8.0)
      tilt
    haml-rails (2.0.1)
      actionpack (>= 5.1)
      activesupport (>= 5.1)
      haml (>= 4.0.6, < 6.0)
      html2haml (>= 1.0.1)
      railties (>= 5.1)
    haml_lint (0.37.0)
      haml (>= 4.0, < 5.3)
      parallel (~> 1.10)
      rainbow
      rubocop (>= 0.50.0)
      sysexits (~> 1.1)
    html2haml (2.2.0)
      erubis (~> 2.7.0)
      haml (>= 4.0, < 6)
      nokogiri (>= 1.6.0)
      ruby_parser (~> 3.5)
    http (4.4.1)
      addressable (~> 2.3)
      http-cookie (~> 1.0)
      http-form_data (~> 2.2)
      http-parser (~> 1.2.0)
    http-cookie (1.0.3)
      domain_name (~> 0.5)
    http-form_data (2.3.0)
    http-parser (1.2.3)
      ffi-compiler (>= 1.0, < 2.0)
    i18n (1.8.7)
      concurrent-ruby (~> 1.0)
    json (2.5.1)
    liquid (4.0.3)
    listen (3.4.1)
      rb-fsevent (~> 0.10, >= 0.10.3)
      rb-inotify (~> 0.9, >= 0.9.10)
    loofah (2.9.0)
      crass (~> 1.0.2)
      nokogiri (>= 1.5.9)
    mail (2.7.1)
      mini_mime (>= 0.1.1)
    mailosaur (7.0.2)
      faraday (>= 0.9.0, <= 1.0)
      json (>= 1.7.5, <= 3.0)
    marcel (0.3.3)
      mimemagic (~> 0.3.2)
    meta-tags (2.14.0)
      actionpack (>= 3.2.0, < 6.2)
    method_source (1.0.0)
    middleware (0.1.0)
    mime-types (3.3.1)
      mime-types-data (~> 3.2015)
    mime-types-data (3.2020.1104)
    mimemagic (0.3.5)
    mini_mime (1.0.2)
    mini_portile2 (2.5.0)
    minitest (5.14.3)
    multi_json (1.15.0)
    multi_test (0.1.2)
    multipart-post (2.1.1)
    nio4r (2.5.4)
    nokogiri (1.11.1)
      mini_portile2 (~> 2.5.0)
      racc (~> 1.4)
    parallel (1.20.1)
    parser (3.0.0.0)
      ast (~> 2.4.1)
    protobuf-cucumber (3.10.8)
      activesupport (>= 3.2)
      middleware
      thor
      thread_safe
    public_suffix (4.0.6)
    puma (5.1.1)
      nio4r (~> 2.0)
    racc (1.5.2)
    rack (2.2.3)
    rack-proxy (0.6.5)
      rack
    rack-test (1.1.0)
      rack (>= 1.0, < 3)
    rails (6.0.3.4)
      actioncable (= 6.0.3.4)
      actionmailbox (= 6.0.3.4)
      actionmailer (= 6.0.3.4)
      actionpack (= 6.0.3.4)
      actiontext (= 6.0.3.4)
      actionview (= 6.0.3.4)
      activejob (= 6.0.3.4)
      activemodel (= 6.0.3.4)
      activerecord (= 6.0.3.4)
      activestorage (= 6.0.3.4)
      activesupport (= 6.0.3.4)
      bundler (>= 1.3.0)
      railties (= 6.0.3.4)
      sprockets-rails (>= 2.0.0)
    rails-dom-testing (2.0.3)
      activesupport (>= 4.2.0)
      nokogiri (>= 1.6)
    rails-html-sanitizer (1.3.0)
      loofah (~> 2.3)
    railties (6.0.3.4)
      actionpack (= 6.0.3.4)
      activesupport (= 6.0.3.4)
      method_source
      rake (>= 0.8.7)
      thor (>= 0.20.3, < 2.0)
    rainbow (3.0.0)
    rake (13.0.3)
    rb-fsevent (0.10.4)
    rb-inotify (0.10.1)
      ffi (~> 1.0)
    react-rails (2.6.1)
      babel-transpiler (>= 0.7.0)
      connection_pool
      execjs
      railties (>= 3.2)
      tilt
    redis (4.2.5)
    regexp_parser (1.8.2)
    retriable (3.1.2)
    rexml (3.2.4)
    rich_text_renderer (0.2.2)
    rspec (3.10.0)
      rspec-core (~> 3.10.0)
      rspec-expectations (~> 3.10.0)
      rspec-mocks (~> 3.10.0)
    rspec-core (3.10.1)
      rspec-support (~> 3.10.0)
    rspec-expectations (3.10.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.10.0)
    rspec-mocks (3.10.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.10.0)
    rspec-rails (4.0.2)
      actionpack (>= 4.2)
      activesupport (>= 4.2)
      railties (>= 4.2)
      rspec-core (~> 3.10)
      rspec-expectations (~> 3.10)
      rspec-mocks (~> 3.10)
      rspec-support (~> 3.10)
    rspec-support (3.10.1)
    rubocop (1.0.0)
      parallel (~> 1.10)
      parser (>= 2.7.1.5)
      rainbow (>= 2.2.2, < 4.0)
      regexp_parser (>= 1.8)
      rexml
      rubocop-ast (>= 0.6.0)
      ruby-progressbar (~> 1.7)
      unicode-display_width (>= 1.4.0, < 2.0)
    rubocop-ast (1.4.0)
      parser (>= 2.7.1.5)
    ruby-progressbar (1.11.0)
    ruby_parser (3.15.1)
      sexp_processor (~> 4.9)
    rubyzip (2.3.0)
    selenium-webdriver (4.0.0.alpha6)
      childprocess (>= 0.5, < 4.0)
      rubyzip (>= 1.2.2)
      websocket (~> 1.0)
    semantic_range (2.3.1)
    sexp_processor (4.15.2)
    simplecov (0.21.2)
      docile (~> 1.1)
      simplecov-html (~> 0.11)
      simplecov_json_formatter (~> 0.1)
    simplecov-html (0.12.3)
    simplecov_json_formatter (0.1.2)
    site_prism (3.7)
      addressable (~> 2.5)
      capybara (~> 3.8)
      site_prism-all_there (>= 0.3.1, < 1.0)
    site_prism-all_there (0.3.2)
    spring (2.1.1)
    spring-watcher-listen (2.0.1)
      listen (>= 2.7, < 4.0)
      spring (>= 1.2, < 3.0)
    sprockets (4.0.2)
      concurrent-ruby (~> 1.0)
      rack (> 1, < 3)
    sprockets-rails (3.2.2)
      actionpack (>= 4.0)
      activesupport (>= 4.0)
      sprockets (>= 3.0.0)
    sys-uname (1.2.2)
      ffi (~> 1.1)
    sysexits (1.2.0)
    temple (0.8.2)
    thor (1.0.1)
    thread_safe (0.3.6)
    tilt (2.0.10)
    turbolinks (5.2.1)
      turbolinks-source (~> 5.2)
    turbolinks-source (5.2.0)
    tzinfo (1.2.9)
      thread_safe (~> 0.1)
    unf (0.1.4)
      unf_ext
    unf_ext (0.0.7.7)
    unicode-display_width (1.7.0)
    view_component (2.24.0)
      activesupport (>= 5.0.0, < 7.0)
    web-console (4.1.0)
      actionview (>= 6.0.0)
      activemodel (>= 6.0.0)
      bindex (>= 0.4.0)
      railties (>= 6.0.0)
    webdrivers (4.4.2)
      nokogiri (~> 1.6)
      rubyzip (>= 1.3.0)
      selenium-webdriver (>= 3.0, < 4.0)
    webpacker (5.2.1)
      activesupport (>= 5.2)
      rack-proxy (>= 0.6.1)
      railties (>= 5.2)
      semantic_range (>= 2.3.0)
    websocket (1.2.9)
    websocket-driver (0.7.3)
      websocket-extensions (>= 0.1.0)
    websocket-extensions (0.1.5)
    xpath (3.2.0)
      nokogiri (~> 1.8)
    zeitwerk (2.4.2)

PLATFORMS
  ruby

DEPENDENCIES
  capybara (~> 3.8)
  climate_control
  contentful
  cucumber (~> 4.0)
  cucumber-rails (~> 2.0)
  haml-rails
  haml_lint
  liquid (= 4.0.3)
  listen
  mailosaur (~> 7.0)
  meta-tags (~> 2.14)
  puma
  rails (~> 6.0.3)
  react-rails
  redis
  retriable (~> 3.1)
  rich_text_renderer
  rspec
  rspec-rails
  selenium-webdriver (= 4.0.0.alpha6)
  simplecov
  site_prism (~> 3.7)
  spring
  spring-watcher-listen
  turbolinks
  tzinfo-data
  view_component
  web-console
  webdrivers (~> 4.4)
  webpacker

RUBY VERSION
   ruby 2.7.2p137

BUNDLED WITH
   2.1.4

Which command did you run?

bundle update --conservative

What were you expecting to happen?

All gems in my gemfile to be updated, all dependencies not to be updated unless they were directly required to

What actually happened?

Extra updates happened. Below is the git diff

       zeitwerk (~> 2.2, >= 2.2.2)
     addressable (2.7.0)
       public_suffix (>= 2.0.2, < 5.0)
-    ast (2.4.1)
+    ast (2.4.2)
     babel-source (5.8.35)
     babel-transpiler (0.7.0)
       babel-source (>= 4.0, < 6)
       execjs (~> 2.0)
     bindex (0.8.1)
     builder (3.2.4)
-    capybara (3.34.0)
+    capybara (3.35.0)
       addressable
       mini_mime (>= 0.1.3)
       nokogiri (~> 1.8)
       rack (>= 1.6.0)
       rack-test (>= 0.6.3)
-      regexp_parser (~> 1.5)
+      regexp_parser (>= 1.5, < 3.0)
       xpath (~> 3.2)
     childprocess (3.0.0)
     climate_control (0.2.0)
-    concurrent-ruby (1.1.7)
+    concurrent-ruby (1.1.8)
     connection_pool (2.2.3)
     contentful (2.15.4)
       http (> 0.8, < 5.0)
@@ -118,7 +118,7 @@ GEM
       cucumber-core (~> 7.1, >= 7.1.0)
       cucumber-cucumber-expressions (~> 10.1, >= 10.1.0)
       cucumber-messages (~> 12.2, >= 12.2.0)
-    diff-lcs (1.4.4)
+    diff-lcs (1.3)
     docile (1.3.5)
     domain_name (0.5.20190701)
       unf (>= 0.0.5, < 1.0.0)
@@ -175,7 +175,7 @@ GEM
       nokogiri (>= 1.5.9)
     mail (2.7.1)
       mini_mime (>= 0.1.1)
-    mailosaur (7.0.2)
+    mailosaur (7.1.0)
       faraday (>= 0.9.0, <= 1.0)
       json (>= 1.7.5, <= 3.0)
     marcel (0.3.3)
@@ -253,7 +253,7 @@ GEM
       railties (>= 3.2)
       tilt
     redis (4.2.5)
-    regexp_parser (1.8.2)
+    regexp_parser (2.0.3)
     retriable (3.1.2)
     rexml (3.2.4)
     rich_text_renderer (0.2.2)
@@ -278,16 +278,16 @@ GEM
       rspec-mocks (~> 3.10)
       rspec-support (~> 3.10)
     rspec-support (3.10.1)
-    rubocop (1.0.0)
+    rubocop (1.8.1)
       parallel (~> 1.10)
-      parser (>= 2.7.1.5)
+      parser (>= 3.0.0.0)
       rainbow (>= 2.2.2, < 4.0)
-      regexp_parser (>= 1.8)
+      regexp_parser (>= 1.8, < 3.0)
       rexml
-      rubocop-ast (>= 0.6.0)
+      rubocop-ast (>= 1.2.0, < 2.0)
       ruby-progressbar (~> 1.7)
-      unicode-display_width (>= 1.4.0, < 2.0)
-    rubocop-ast (1.4.0)
+      unicode-display_width (>= 1.4.0, < 3.0)
+    rubocop-ast (1.4.1)
       parser (>= 2.7.1.5)
     ruby-progressbar (1.11.0)
     ruby_parser (3.15.1)
@@ -325,7 +325,7 @@ GEM
       ffi (~> 1.1)
     sysexits (1.2.0)
     temple (0.8.2)
-    thor (1.0.1)
+    thor (1.1.0)
     thread_safe (0.3.6)
     tilt (2.0.10)
     turbolinks (5.2.1)
@@ -336,15 +336,15 @@ GEM
     unf (0.1.4)
       unf_ext
     unf_ext (0.0.7.7)
-    unicode-display_width (1.7.0)
-    view_component (2.24.0)
+    unicode-display_width (2.0.0)
+    view_component (2.25.0)
       activesupport (>= 5.0.0, < 7.0)
     web-console (4.1.0)
       actionview (>= 6.0.0)
       activemodel (>= 6.0.0)
       bindex (>= 0.4.0)
       railties (>= 6.0.0)
-    webdrivers (4.4.2)
+    webdrivers (4.5.0)
       nokogiri (~> 1.6)
       rubyzip (>= 1.3.0)
       selenium-webdriver (>= 3.0, < 4.0)

Gem updates and expectations
ast 2.4.1 -> 2.4.2. - UNEXPECTED
capybara 3.34.0 -> 3.35.0 - EXPECTED (This changed regexp_parser's restrictions, but still shouldn't edit it)
concurrent-ruby 1.1.7 -> 1.1.8 - UNEXPECTED
mailosaur 7.0.2 -> 7.1.0 - EXPECTED
regexp_parser - 1.8.2 -> 2.0.3 - UNEXPECTED
rubocop - 1.0.0 -> 1.8.1 - EXPECTED (This changed regexp_parser, rubocop-ast and unicode_display_width restrictions, but still shouldn't edit any of them, as their previous versions were valid)
rubocop-ast 1.4.0 -> 1.4.1 - UNEXPECTED
thor 1.0.1 -> 1.1.0 - UNEXPECTED
unicode-display_width 1.7.0 -> 2.0.0 - UNEXPECTED
view_component - 2.24.0 -> 2.25.0 - UNEXPECTED webdrivers` 4.4.2 -> 4.5.0 - EXPECTED

If not included with the output of your command, run bundle env and paste the output below

Was the same as above basically

[deivid-rodriguez]

Hi @luke-hill! Thanks for reporting this issue!

I totally agree with your expectations. The --conservative option should be... well, conservative.

I looked at the --conservative documentation and it seems that the current behaviour is somehow expected? Apparently what it's supposed to do is only to avoid updating transitive dependencies if they are also transitive dependencies of another top level package.

It's definitly worth digging into history to see why this option was introduced, and why it was introduced with this particular meaning, but my initial take is to change its behavior to what everyone intuitively expects: update only top-level dependencies.

[deivid-rodriguez]

After reading through rubygems/bundler-features#122 and rubygems/bundler#4980, I don't see anything in there suggesting this is actually expected, except for this one line in the commit message, and the relative "unlock shared dependencies" naming:

The internal flag is more descriptive as to what actually takes place: It locks any shared dependencies from the gem(s) being updated.

Just to confirm that I'll be treating this as a plain bug without any backwards compatibility concerns.

[deivid-rodriguez]

Hei @luke-hill!

I have a question for you. Did you maybe edit the lockfile directly to share this repro case? I'm asking because you shared a lockfile that includes an incorrect resolution 😬. I really hope bundler is not generating this lockfile automatically 🤞.

The error in the lockfile is that it includes diff-lcs 1.4.4 and also cucumber-4.1.0, which depends on diff-lcs >= 1.3, ~> 1.3, < 1.4.

[luke-hill]

Yep I did. So if there's one or two inconsistencies ignore it.

The OP stands. There was some stuff that I ran that shouldn't be committed so I removed it.

[deivid-rodriguez]

That's great to hear. I created #4459 to warn this situation.

[deivid-rodriguez]

@luke-hill It took me a while to get to this, but #4692 should fix this!

[luke-hill]

Around the time this went in was around the time I was switching jobs / on holiday. So forgot to say thanks / well done. :) Top work!