Bundler auto-update during bundle install uses rubygems.org instead custom gem server

[lxxxvi]

Hello dear maintainers! 👋

I think I have found a thing, maybe a bug... it maybe related to #6405 too.

Thanks a lot for your time and help, I appreciate it!

Describe the problem as clearly as you can

We'd like to do bundle install in a corporate network where there's no access to the internet.
We do have our own gem server which is configured in the Gemfile and in gemrc:

$ cat Gemfile | grep source
source "https://example.com/"

$ gem env
RubyGems Environment:
  # ...
  - GEM CONFIGURATION:
  # ...
     - :sources => ["https://example.com/"]
  - REMOTE SOURCES:
     - https://example.com/
  # ...

When doing bundle install we see this kind of output:

$ bundle install
Bundler 2.4.13 is running, but your lockfile was generated with 2.4.19. Installing Bundler 2.4.19 and restarting using that version.
Fetching source index from https://rubygems.org/

Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from https://rubygems.org/ due to underlying error <Net::OpenTimeout: Net::OpenTimeout (https://rubygems.org/specs.4.8.gz)>

(Basically the same happens with bundle update --bundler)

The error is displayed 4 times, each taking 60 seconds. After that it continues with the actual gem installation against the (correctly) configured gem server (https://example.com).

So, that means that bundle install works, but it wastes 4 minutes trying to update bundler which is attempted to be fetched from https://rubygems.org instead of https://example.com.

I presume this is because https://rubygems.org is hardcoded here (?!):

rubygems/bundler/lib/bundler/self_manager.rb

Lines 119 to 126 in 313bf1e

	def remote_specs
	@remote_specs ||= begin
	source = Bundler::Source::Rubygems.new("remotes" => "https://rubygems.org")
	source.remote!
	source.add_dependency_names("bundler")
	source.specs
	end
	end

Did you try upgrading rubygems & bundler?

The problem occurs during updating bundler, which doesn't work in a private network and a custom gem server setting.

Post steps to reproduce the problem

Run bundle install with an outdated bundler version, in private network and with a custom gem server.

Which command did you run?

bundle install

What were you expecting to happen?

It should install the gems with a pre-step to update bundler, everything against the custom gem server.

What actually happened?

bundle install ran into timeouts while trying to update bundle.

If not included with the output of your command, run bundle env and paste the output below

(n/a)

[duckinator]

Hey, sorry for the lack of response here. I'm hoping to look into this in the next few weeks.

I suspect using a specific server for Bundler is to avoid installing the wrong gem entirely (e.g. if a server has another gem called Bundler), but I'm not sure. I'll look into it.

If nothing else, we should at least add a way to opt out of this functionality entirely. While this feature helps in many cases, it's obviously causing problems in others, and the least we can do is provide a way to disable it.

A workaround would be to manually install the version you want, then run bundle _X.Y.Z_ install. E.g., for 2.4.21 you'd install that version then do bundle _2.4.21_ install.

[lxxxvi]

Thank you for looking into this and your feedback, I appreciate it! 🙌

I suspect using a specific server for Bundler is to avoid installing the wrong gem entirely (e.g. if a server has another gem called Bundler), but I'm not sure. I'll look into it.

That's an interesting case, but even then I'd still expect it to use the REMOTE_SOURCE. 🤷

If nothing else, we should at least add a way to opt out of this functionality entirely. While this feature helps in many cases, it's obviously causing problems in others, and the least we can do is provide a way to disable it.

That would be nice, yes.

[duckinator]

Thinking about this again, I came up with a potential solution:

If the list of sources doesn't include rubygems.org, disable auto-updates.

This means that by default the experience is nicer AND in environments configured to not touch rubygems.org, we don't violate that expectation.

Looks like the opt-out was added by #6817, and you can use that as a workaround.

[deivid-rodriguez]

I think I'd instead use whatever the default Gemfile source is for downloading Bundler? That's what an explicit bundle update --bundler would use anyways.

[duckinator]

@deivid-rodriguez I was under the impression that using rubygems.org for updating Bundler was intentional. If that's not the case, yeah, it using whatever the default Gemfile source is makes sense. 👍

[deivid-rodriguez]

Nah, it was just me not properly considering all use cases when we shipped this feature.

[deivid-rodriguez]

Alternatively, and easier to achieve, we can look at Gem.sources and pick whatever it's there if it's a single element, and keep defaulting to rubygems.org otherwise.

Looking into the Gemfile is trickier, because we'd need to parse the Gemfile with the current bundle version but there's chances that the Gemfile uses some feature not supported by the running Bundler, which is the whole point of the auto-install feature.