Extend Gemfile override DSL with :all target and metadata fields (Phase 2)#9530
Merged
Extend Gemfile override DSL with :all target and metadata fields (Phase 2)#9530
override DSL with :all target and metadata fields (Phase 2)#9530Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Extends Bundler’s Gemfile override DSL (introduced in Phase 1) to support metadata constraint overrides (required_ruby_version / required_rubygems_version), a global :all target for metadata fields, and ensures these overrides are honored both during dependency resolution and install-time compatibility checks.
Changes:
- Add metadata-aware override application across resolution, lockfile behavior, and install-time checks, including support for
override :all, required_*_version: .... - Introduce override propagation through
SpecSet/LazySpecificationand add resolve-failure diagnostics listing active overrides with Gemfile locations. - Update documentation/manpages and add broad test coverage for the new DSL surface and behaviors.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| spec/install/gemfile/override_spec.rb | Integration coverage for :all, metadata overrides, lockfile behavior, diagnostics, and install-time compatibility. |
| spec/bundler/spec_set_spec.rb | Unit tests for SpecSet#with_overrides behavior and override propagation. |
| spec/bundler/override_spec.rb | Unit coverage for Override.find_for and override-aware metadata matching helpers. |
| spec/bundler/dsl_spec.rb | DSL parsing/validation tests for new fields, :all, atomic rejection, and plugin warning behavior. |
| bundler/lib/bundler/spec_set.rb | Add override storage/propagation and use override-aware metadata validation during spec selection. |
| bundler/lib/bundler/resolver.rb | Apply metadata overrides to metadata dependencies and append override diagnostics on solve failure. |
| bundler/lib/bundler/override.rb | Add find_for (per-gem with :all fallback) and record Gemfile source locations. |
| bundler/lib/bundler/match_remote_metadata.rb | Ensure remote metadata fields are loaded for override-aware matching methods. |
| bundler/lib/bundler/match_metadata.rb | Add override-aware Ruby/RubyGems compatibility checks via “effective requirement” computation. |
| bundler/lib/bundler/man/gemfile.5.ronn | Document :all and metadata fields, install-time behavior, diagnostics, and plugin redundancy. |
| bundler/lib/bundler/man/gemfile.5 | Generated manpage updates corresponding to the ronn changes. |
| bundler/lib/bundler/lazy_specification.rb | Make choose_compatible honor overrides when filtering incompatible candidates. |
| bundler/lib/bundler/installer.rb | Honor overrides during install-time Ruby/RubyGems compatibility checks. |
| bundler/lib/bundler/dsl.rb | Extend supported override fields, validate operations, forbid :all + version, and warn about redundant plugin. |
| bundler/lib/bundler/definition.rb | Thread overrides through locked/resolved SpecSets and refine unlock behavior around metadata vs :all. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Definition#apply_override_to, Definition#converge_dependencies, and Resolver#apply_overrides each duplicated the same find expression. Centralizing it on Override prepares for upcoming fields (and the :all target) without repeating the predicate in every call site. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before this change, `override "rails", version: "not a version"` was accepted by Bundler::Dsl#override and only failed later when the resolver tried to instantiate Gem::Requirement. Surface the error at the Gemfile evaluation step so the user sees it on the offending line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lock the byroot policy decision (overrides are a Gemfile concept and must not be serialized into Gemfile.lock) with a regression test, so a future change that starts emitting override metadata in the lock would fail loudly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extend the override DSL whitelist so per-gem metadata fields can be declared. The :all target is rejected for now with a "not yet supported" error so the field is purely per-gem until the next step adds :all propagation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a spec's runtime dependencies are gathered for the resolver, its required_ruby_version / required_rubygems_version metadata flow as synthetic Ruby\0 / RubyGems\0 dependencies. Rewrite those before they reach the dependency hash so per-gem overrides on those fields take effect during resolution. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-dep loop in converge_dependencies only knows about version: overrides via apply_override_to + matches_spec?. Metadata overrides on direct deps were therefore invisible to lockfile change detection and would silently no-op against an existing lock. Extend converge_overrides_outside_dependencies to also unlock direct deps when the override targets a non-version field. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add integration coverage exercising :ignore_upper and nil overrides against per-gem metadata fields, transitive propagation, and lockfile re-resolution when a metadata override is added against an existing lockfile. The cases drive `bundle lock` so they exercise the resolver without RubyGems' install-time required_ruby_version gate, which is addressed in a later step. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove the temporary "not yet supported" guard so a Gemfile may write `override :all, required_ruby_version: :ignore_upper` and similar forms. The version: ban for :all stays — version requirements are inherently per-gem. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Override.find_for now returns the per-gem entry when present and otherwise the matching :all entry on the same field. This is the single dispatch point for overrides in Definition and Resolver, so the fallback is what wires :all into resolution and lockfile change detection without further plumbing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
An :all override applies to every gem's metadata, so we have no way to know which locked entries it affects without re-resolving. Force-unlock them all when an :all override appears so the resolver gets a fresh chance to apply the override. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Exercise :all required_ruby_version overrides applied to multiple gems at once, the per-gem precedence rule, and re-resolution against an existing lockfile when an :all override is introduced. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
matches_current_ruby? / matches_current_rubygems? now look up the current Definition's overrides via Bundler.overrides and apply them before checking against the runtime Ruby/RubyGems version. This covers Installer#ensure_specs_are_compatible! and the materialize- layer choose_compatible / SpecSet#valid? checks uniformly without plumbing overrides through every materialization site. When no Definition is set yet (e.g. RubyGems-side calls outside a Bundler.definition block), Bundler.overrides returns an empty list and the methods fall through to their original behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drive bundle install end-to-end with a gem whose required_ruby_version or required_rubygems_version excludes the current runtime, asserting that a per-gem override (and an :all override) makes the install succeed instead of erroring at the install-time compatibility gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bundler::Dsl#override now records caller_locations(1, 1).first on each Override so the originating Gemfile line can be surfaced in later diagnostics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Append the list of currently active overrides (with Gemfile location, when known) to the SolveFailure message so a user investigating a "could not find compatible versions" error sees what override changed the constraint set instead of being misled by the resolver-reported requirement. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Update the OVERRIDE section to cover the :all target, the required_ruby_version / required_rubygems_version fields, and the diagnostic shown on resolve failure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2.C wired overrides into MatchMetadata via Bundler.overrides, a process-wide accessor read every time a spec answered matches_current_metadata?. That leaked the user's Gemfile overrides into Bundler-internal callers like SelfManager#remote_specs, where overrides have no business: a Gemfile override could let bundle self-update consider Bundler releases that are actually incompatible with the running Ruby/RubyGems. Revert MatchMetadata's matches_current_ruby? and matches_current_rubygems? to evaluate the spec's own metadata, and add explicit matches_current_*_with_overrides? variants. Pass overrides explicitly: Installer#ensure_specs_are_compatible! gets them from @Definition, LazySpecification#choose_compatible reads its newly-added @OVERRIDES attribute, and SpecSet#valid? reads @OVERRIDES set on the SpecSet. Definition propagates @OVERRIDES to the SpecSets it constructs and to the LazySpecs they contain. SelfManager and other callers that should keep evaluating real gemspec metadata reach the strict path unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously, any :all override called unlock_all_locked_specs_for_override which pushed every locked spec into @gems_to_unlock. A user adding a narrow `override :all, required_ruby_version: :ignore_upper` thus paid for a full re-resolve that could pull unrelated dependency upgrades/downgrades. Make :all overrides leave the lockfile alone at converge time. They take effect on a fresh resolution (no lockfile) or when the user opts in via `bundle update`. Per-gem overrides retain their unlock for the named gem since the user explicitly named the target. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Definition#resolve falls back to an existing lockfile when nothing about the Gemfile or the locked deps changed. The two SpecSet rebuild paths (deleted_deps subset and the redundant-platform-specific-gems fallback) constructed fresh SpecSet instances without carrying @OVERRIDES forward, so any LazySpec produced from them lost its override context. After Step G that mattered: an :all metadata override does not pre-unlock anything by design, which means it must flow through these reuse paths intact. Without it, the materialize layer either silently re-resolved (which churns the lockfile) or, on the install-time check, fell back to the spec's strict required_ruby_version metadata. Calling with_overrides on both rebuilt SpecSets keeps the install-time behavior consistent across resolve and lockfile-reuse paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpecSet#incomplete_specs_for_platform constructed a fresh self.class.new(@specs) for platform validation but never copied @OVERRIDES. Platform-validity decisions therefore evaluated strict required_ruby_version / required_rubygems_version metadata even when resolution was running with overrides, so a metadata override could allow a gem everywhere except platform validation, where the platform might be marked incomplete and pruned. Carry @OVERRIDES forward via with_overrides on the cloned SpecSet. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpecSet#with_overrides cascaded into each contained spec via `respond_to?(:overrides=)`. RemoteSpecification#respond_to? forwards to _remote_specification, which materializes the backing gemspec just to answer the predicate. spec/runtime/require_spec.rb verifies that Bundler does not load gemspecs it does not need by deliberately poisoning one with `raise 'broken gemspec'`; the cascade tripped that guard and made `Bundler.setup` blow up. Gate the cascade on `is_a?(LazySpecification)` instead. Only LazySpecification declares `attr_accessor :overrides` (used by `#choose_compatible`), so the predicate is equivalent for any spec we ever set overrides on, and it never triggers the lazy gemspec load. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpecSet previously kept its own @OVERRIDES and a with_overrides setter that had to be chained on every SpecSet.new(...) site (~13 sites in Definition alone). Two Codex review rounds both flagged forgotten chains in different SpecSet construction paths, which is exactly the class of bug the chain pattern invites: it is purely "remember to write" with no compiler help. Move the override list to LazySpecification#overrides instead. The LazySpec is the natural carrier — it is the value object the resolver and install paths already pass around, and choose_compatible already read overrides off it. Override.attach(specs, overrides) is added as the dual of Override.find_for so Definition (after lockfile load) and Resolver (after solve_versions) can populate the overrides on every LazySpec they hand out, and LazySpecification.from_spec carries the list forward when one LazySpec spawns another. Generic spec types (StubSpecification, plain Gem::Specification, RemoteSpecification) are intentionally ignored so generic metadata callers (SelfManager, materialize-time strict checks) keep their current strict semantics. SpecSet drops attr_accessor :overrides, the @OVERRIDES initialisation, the with_overrides cascade, and reverts SpecSet#valid? to the strict matches_current_metadata? check. Every SpecSet.new(...) site in Definition stops chaining .with_overrides — the LazySpecs already carry the context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
complete_platform validates platform-specific candidates returned by spec.source.specs.search, which are remote specs that do not carry the override list. Borrow the override list from the LazySpec exemplar already in scope so platform-variant validation uses the same effective metadata as the install/resolve path. Also propagate the overrides onto the synthesized LazySpec built from platform_spec. Without this, the next complete_platform call could pick the synthesized variant as its exemplar (it is now in the set returned by lookup) and fall back to strict matching, dropping platforms that the user's override would otherwise allow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was the end-user or developer problem that led to this PR?
Follow-up to #9517 (Phase 1), continuing #9494. Phase 1 only covered per-gem
version:overrides; Phase 2 adds the remaining operations the discussion called for: an:alltarget and therequired_ruby_version/required_rubygems_versionmetadata fields, with overrides honored at install time as well as during resolution.What is your fix for the problem, implemented in this PR?
Extends the existing
overrideDSL with metadata fields and an:alltarget.:all + version:stays banned — version requirements are inherently per-gem. A per-gem override takes precedence over an:alloverride on the same field.The Phase 1 operations carry over unchanged: an absolute requirement string replaces the constraint,
:ignore_upperdrops</<=and folds~>into>=, andnilcollapses to>= 0.Overrides are now honored by the install-time Ruby / RubyGems compatibility check too (previously install would still error even when resolution had picked a deliberately overridden gem). When resolution still fails, Bundler appends the active overrides — with their Gemfile location — to the error message so it's clear which directive shaped the constraint set.
Make sure the following tasks are checked