Avoiding "already initialized constant errors" with net/protocol

[stanhu]

If I have a gem that depends on net/imap v0.2.1, I see that it conflicts with the Ruby interpreter's version:

$ bundle exec rspec spec
/home/travis/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/protocol.rb:66: warning: already initialized constant Net::ProtocRetryError
/home/travis/.rvm/gems/ruby-2.6.6/gems/net-protocol-0.1.0/lib/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/home/travis/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/protocol.rb:206: warning: already initialized constant Net::BufferedIO::BUFSIZE
/home/travis/.rvm/gems/ruby-2.6.6/gems/net-protocol-0.1.0/lib/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/home/travis/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/protocol.rb:503: warning: already initialized constant Net::NetPrivate::Socket
/home/travis/.rvm/gems/ruby-2.6.6/gems/net-protocol-0.1.0/lib/net/protocol.rb:503: warning: previous definition of Socket was here

Is there a way this updated version of the gem can be used as a separate dependency without generating these warnings? https://stackoverflow.com/a/9117903/1992201 has some suggestions, but these still seem like hacks. Does it make sense to remove net/protocol as a hard dependency? MRI appears to have an exact copy of protocol.rb.

[shugo]

net/protocol and net/imap have been default gems since Ruby 3.0, so it may introduce problems to remove net-protocol from dependencies.
On Ruby 2.6, net/imap itself is not a (default) gem, so it may be hard to use the net-imap gem without hacks.

@hsbt What do you think?

[deivid-rodriguez]

@stanhu I'm interested in this is issue. Can you share steps to reproduce the warnings?

[stanhu]

@deivid-rodriguez The easiest way I've been able to reproduce it:

git clone git@github.com:tpitale/mail_room.git
cd mail_room
echo 2.7.2 > .ruby_version # Choose your version
bundle install
bundle exec rspec spec/lib/mailbox_spec.rb

[deivid-rodriguez]

Thanks, I'll have a look!

[deivid-rodriguez]

The proper fix here in my opinion would be that the faraday-net_http gem would declare the net-http gem as a dependency.

This is the explanation of what's happening:

• When the faraday-net_http gem is required, it then requires net/http.
• Then the stdlib version of net/http is loaded, since bundler doesn't know net-http is a gem dependency, so it doesn't add the gemified version to to the $LOAD_PATH.
• Then the stdlib version of net/http requires net/protocol relatively, since that's what it used to happen before these gems were gemified. See ruby/net-http@6da598e. So the stdlib version of net/protocol is loaded.
• Then when net-imap is required, it also uses net-protocol but this time the gemified version of net-protocol is used, because net-imap properly declares it as a dependency, so bundler knows about it and has put it in the $LOAD_PATH. This is what causes the double load of net/protocol.

If faraday-net_http explicitly declares net-http as a dependency, then the gemified version of all gems involved will be consistently loaded and there will be no redefinition warnings.

I suggest that you proposed this update to faraday-net_http and in the mean time, you can workaround this issue by adding gem "net-http" to your Gemfile.

[stanhu]

@deivid-rodriguez Thanks for that tip! That seems to work well. I've submitted lostisland/faraday-net_http#5 and tpitale/mail_room#123.

[stanhu]

One issue is that net-http requires Ruby 2.6, so older Rubies won't work with this add_dependency.

[hsbt]

It's not related dependencies of gemspec.

[deivid-rodriguez]

One issue is that net-http requires Ruby 2.6, so older Rubies won't work with this add_dependency.

Oh, I see, that's unfortunate. I guess you need to workaround this via Gemfile until other dependencies also drop Ruby 2.5 support :(. For what it's worth, ruby 2.5 was dropped at ruby/net-http@e6ab6d7. My guess is that the only reason was that CI didn't pass.

[deivid-rodriguez]

Cointidentially I run into a similar instance of this issue myself today. To make things easier for gem authors and motivate them to add the dependency, I propose to release at least one gemified version of net-http that supports ruby 2.5. I proposed that at ruby/net-http#19.

[iMacTia]

Hi @deivid-rodriguez, I'd like to chime in on the discussion since I've been asked multiple times to add net-http as a dependency tofaraday-net_http as a result of your comment and I'm not completely sure yet if that's the correct thing to do.

I explained my understanding of the issue in this comment and why I believe it is a mistake to add these gems explicitly in gemspec.

If faraday were to add net-http as a dependency, it would force every project using it to switch to the gemified version of the libray. This would be OK on Ruby 3.0, but it might be a disaster for previous rubies, where the stdlib version is still present.

For example, I'd expect issues like this to pop up if I did that.

I'm not sure why some gems have started adding explicit dependencies to these gems, but it seems like letting Ruby figuring it out by itself (when to use the stdlib or the standard gem) is the best way to avoid this mix and match and support both Rubies older and newer than 3.0.

I understand some people are affected by this issue because they use a gem that has added one of these dependencies to their gemspec, and I can see why it's annoying to now having to add more dependencies themselves.
But as I see it, if I were to add the same dependency to faraday this could actually increase the conflicts rather than improving the situation.

It would be great to discuss this and agree to a solution that could benefit everyone 🙏

[deivid-rodriguez]

Sorry I don't have a lot of time to look into this right now, but I can say something similar to this has been discussed in the upstream Ruby tracker, and there's no agreement. For example, https://bugs.ruby-lang.org/issues/16951. Note that I did propose ruby/net-http#19 after that message you mention to avoid this problem, but it didn't happen.

Not sure about your specific situation and the range of rubies where issues would pop up vs the range of rubies that would benefit from it. If this would only cause problems on rubies that have reached EOL, and those problems are only warnings, then I do think it's worth prioritizing users of supported rubies. But you know best what makes more sense for your libraries, and of course you can chime in the core tracker and propose any solutions. As I said, I don't have a lot of time right now, sorry :(

[iMacTia]

This is great context, thank you @deivid-rodriguez.
Glad to hear there's some discussion ongoing, I'd definitely like to get to the bottom of this before releasing any change in Faraday that could potentially make things even worse

[deivid-rodriguez]

No problem, sorry I can't help more right now, and I appreciate your conciousness and will to get to the bottom of things.

[voxik]

it would force every project using it to switch to the gemified version of the libray.

This is very vague claim and might or might not be true. It very much depends on the context.

it might be a disaster for previous rubies, where the stdlib version is still present.

This is overstatement. Generally, there is nothing wrong in using gemified version of library in old Ruby. The net-http gem supports Ruby 2.6+, so unless you talk about way older versions of Ruby, then it should be just fine.

[deivid-rodriguez]

Right, I'm still unclear about the issue with newer rubies. There's many moving parts so there could be some issue, I just don't know what it is right now.

As I have pointed out in other places, adding explicit dependencies helps Bundler and RubyGems provide consistent results. And also helps users because the exact gem version is recorded in the lockfile just like any other dependency, so you get consistent results when using the dependency even if you switch rubies.

[gepinedo]

The proper fix here in my opinion would be that the faraday-net_http gem would declare the net-http gem as a dependency.
This is the explanation of what's happening:

• When the faraday-net_http gem is required, it then requires net/http.
• Then the stdlib version of net/http is loaded, since bundler doesn't know net-http is a gem dependency, so it doesn't add the gemified version to to the $LOAD_PATH.
• Then the stdlib version of net/http requires net/protocol relatively, since that's what it used to happen before these gems were gemified. See ruby/net-http@6da598e. So the stdlib version of net/protocol is loaded.
• Then when net-imap is required, it also uses net-protocol but this time the gemified version of net-protocol is used, because net-imap properly declares it as a dependency, so bundler knows about it and has put it in the $LOAD_PATH. This is what causes the double load of net/protocol.

If faraday-net_http explicitly declares net-http as a dependency, then the gemified version of all gems involved will be consistently loaded and there will be no redefinition warnings.
I suggest that you proposed this update to faraday-net_http and in the mean time, you can workaround this issue by adding gem "net-http" to your Gemfile.

thank you, I resolved the problem

Thanks, @deivid-rodriguez, it works perfectly!