Make --user-install as default

[hsbt]

Description:

Fixes #1394

---

Tasks:

• Describe the problem / feature
• Write tests
• Write code to solve the problem
• Get code review from coworkers / friends

I will abide by the code of conduct.

[simi]

❓ I'm wondering what has changed your opinion @hsbt. I think you were against this in the initial discussion (#1394 (comment)).

[hsbt]

@simi Yes, But I prefer this idea after thinking with XDG compliance: ruby/ruby#2174

[duckinator]

I am 100% in favor of this change. Are there any known complications with it?

[MSP-Greg]

Re complications, maybe edge cases where one has more than one platform of the same Ruby major/minor version installed.

For instance, I normally work with MinGW builds, and have quite a few gems installed via --user-install. Things get interesting when I switch to an mswin build (different platform)...

[MSP-Greg]

While working on some CI issues, this came to mind.

Re Windows, I don't think AppVeyor or Actions have the user bin folder in ENV['PATH'].

Don't know about Ubuntu and macOS on Travis and Actions. I suspect Actions does not. It could be added, but otherwise...

[eregon]

I'm not sure this is a good idea, it can be less reliable and cause confusing errors if multiple Ruby versions are installed.
For instance, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4 and 2.6.5 will all share the same Gem.user_dir: ~/.gem/ruby/2.6.0. Some of them might have been compiled differently, for instance if one is the system Ruby, or was configured with --enable-shared and some other not (the extensions files are separated for that case, but it causes annoying warnings), or if one was configured with some -D flag and some other not.
I can easily imagine very hard-to-debug segfaults due to this.

If Gem.dir is user-writable (typical for ruby-install/ruby-build/RVM), I think it is far safer/more reliable to use it instead of Gem.user_dir.
I think we should only resort to Gem.user_dir if Gem.dir is not user-writable.

This change is also almost guaranteed to break many usages of Ruby which don't add Gem.user_dir + "/bin" to PATH, which seems fairly common.
It will mean adding ruby_prefix/bin to PATH will never be enough anymore to use a given Ruby installation and executables of installed gems.

[rubyFeedback]

eregon - you need to remember the other trade-off too, in particular on HaikuOS with the
read-only part of the filesystem. I am not at all against sympathy with working with
multiple ruby versions at all, but at the same time you have this problem that ruby can
not be easily used on HaikuOS, and that the initial issue was raised in 2015. Back then
Hiroshi also commented on a few problems he saw in regards to rbenv; others noted
problems with windows (assumingly, permission problems), and you mentioned problems
with multiple versions. None of these resolve the problem on HaikuOS though - in my
opinion the ideal state would be to be able to be as flexible as possible, without
penalizing certain platforms/OS.

I completely disagree with your claim of "difficult segfaults" being a showstopper to this
though. IF there is a segfault then this should be handled differently - --user-install should
never ever be a culprit in this case (how so, anyway?).

I am not at all having a problem with more customization-possibilities, but at the same
time we need to either ask whether niche OS, be it haiku, reactos or anything else, shall
be supported by ruby or not. Mind you: the python support on Haiku is not perfect, but
far better. So do you guys want to see python pushing away ruby completely? The harder
it is for people to use e. g. ruby even on smaller OS, the less likely they will be to use
ruby. So I am all in favour of the changes here, even if there are bumpy rides along the
way. Sometimes you have to move on supporting one new case, even if that requires
fixing other issues along the way.

[duckinator]

Based on the feedback from other folks above, I took a look into how Python, and Haiku, handles things.

---

This change is also almost guaranteed to break many usages of Ruby which don't add Gem.user_dir + "/bin" to PATH, which seems fairly common.
It will mean adding ruby_prefix/bin to PATH will never be enough anymore to use a given Ruby installation and executables of installed gems.

@eregon Python handles this by printing a message like the following:

WARNING: The script emanate is installed in '/packages/python38-3.8.3-1/.self/non-packaged/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

Do you consider something like that a reasonable compromise?

---

@rubyFeedback pip, the package manager for Python, does not default to --user. The issue with RubyGems on Haiku is that it cannot write to /boot/system/lib/ruby/gems/2.7.0, and I believe this could be resolved by the system configuration.

Details for @rubyFeedback

Specifically, I think Haiku may want to use the OS-specific rubygems/defaults/operating_system.rb.

See rubygems/config_file.rb and Fedora 31's /usr/share/rubygems/rubygems/defaults/operating_system.rb.

[simi]

@rubyFeedback Fedora's operating_system.rbis used to modify default paths for system installed Ruby - https://src.fedoraproject.org/rpms/rubygems/blob/master/f/operating_system.rb. Feel free to ping Haiku package maintainers to do the same to fix this problem.

[eregon]

Do you consider something like that a reasonable compromise?

I think that warning would be useful to have.

I still think there is nothing safer than installing gems under the Ruby prefix, if it's writable.
Mixing gems from multiple Ruby versions is risking subtle incompatibilities.
I understand some people might want to risk that, but IMHO it shouldn't be the default as it's more risky.
A trivial example where --user-install does the wrong thing and leads to segfaults is when building ruby master, the same directory will be used for all builds of master, yet the C ABI might change incompatibly between any commit on master.
The ABI for the same minor releases is meant to be compatible, but there are always risks it breaks.

FWIW, chruby will actually set GEM_HOME/GEM_PATH if the Ruby prefix is not writable, but otherwise just write to the Ruby prefix, in part to avoid these potential incompatibilities:
postmodern/chruby#431

[eregon]

For other Ruby managers:
RVM will make sure each Ruby has its own isolated Gem home.
And rbenv will not set anything, meaning it assumes the Ruby prefix is writable.
None of them uses --user-install by default, that doesn't work well and is risky unfortunately.

Adapting operating_system.rb might be a good solution for haiku.

[duckinator]

@eregon I agree that we need to be careful to avoid the problems you mentioned (sorry, just realized I hadn't actually said that earlier, just thought it).

However, revisiting this, I have to wonder: why do 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4 and 2.6.5 all use ~/.gem/ruby/2.6.0? Why not ~/.gem/#{RUBY_ENGINE}/#{RUBY_VERSION} for all Ruby implementations?

As far as I know it would be unique for each Ruby implementation+version combination, and JRuby already appears to do this without issue — I have ~/.gem/jruby/2.5.0, ~/.gem/jruby/2.5.3, ~/.gem/jruby/2.5.7.

This would, in theory, resolve the problem @eregon raised and allow using --user-install. It also should be minimally disruptive: unless I've missed something, the worst case scenario is some people who use an implementation that's neither MRI nor JRuby will have to re-download gems when they otherwise wouldn't have.

---

Regarding @MSP-Greg's remarks about CI: as long as this is implemented in terms of making --user-install the default (like this PR does), you should be able to use --no-user-install and have the behavior be functionally equivalent to the current behavior. If we implement this, it'll be important to document the change and make sure to mention it may impact CI.

(We could also coordinate with the folks who maintain various CI systems or common base containers in an attempt to mitigate those negative effects of this change, but I think that's a separate discussion for after we decide if and how this change would be done.)

[eregon]

@duckinator Currently it's like ~/.gem/#{RUBY_ENGINE}/#{RbConfig::CONFIG['ruby_version']} which uses the "ABI version".
So that makes sense, except that the ABI version is not the only thing capable of making ABI incompatible. Various configure flags can also do that.
Most notably --enable-shared also does that.
RubyGems has some handling for --enable-shared to not segfault but the experience is far from nice.
Imagine you have Ruby 2.6.3 without --enable-shared (default) and Ruby 2.6.5 with --enable-shared (some native extensions require this).
Then you will get many annoying warnings like Ignoring msgpack-1.3.0 because its extensions are not built. Try: gem pristine msgpack --version 1.3.0 for every gem that's installed on 2.6.3 but was not installed on 2.6.5.
It results in half-installed gems, which I think nobody wants to have to deal with.

My experience as a Ruby implementor and contributor to all 3 major Ruby managers is: it's a bad idea to share gems (at least for gems with native extensions).
So I think it should always be opt-in, because it's generally unsafe/confusing/causing many subtle problems, and causes new complicated situations like the one just described above.
The only case it's safe is if there is only one Ruby version using it, but then it's not shared, and there might be a new Ruby installation later that will make it unsafe/confusing.

My recommendation is if you want to install gems as a user, then install Ruby as a user too, which is what most Rubyists do already. If you want for some reason to have Ruby installed in system directories, then you have to choose: do you want to share gems too between multiple users (then you need sudo or some other rights to write to that shared location), or not (then you can use --user-install and accept the risks mentioned above, or set your own GEM_HOME e.g. by application)?

[eregon]

Anyway, if we somehow change the default in RubyGems, we should only not use the default Gem home if it's not writable. If it's writable, we should use the default gem home, because it's safe, convenient, guaranteed to be in PATH, etc.

[voxik]

My recommendation is if you want to install gems as a user, then install Ruby as a user too, which is what most Rubyists do already.

With my Fedora/RHEL maintainer hat, I disagree with this. If people are using Ruby distributed with OS, the --user-install should be the default, otherwise Ruby is unusable and they basically don't have other option the to user install Ruby. That is what we do anyway, but we need to change defaults which is suboptimal as well.

[MSP-Greg]

I don't have strong feelings about this, as long whatever the default can be changed. The one question I have is what will Bundler do? At present, with no path changes, I believe it always installs to Gem.default_dir, regardless of settings in .gemrc. Will it now install in Gem.user_dir?

[duckinator]

My experience as a Ruby implementor and contributor to all 3 major Ruby managers is: it's a bad idea to share gems (at least for gems with native extensions).

Yes, I 100% agree with this. I think what I'm asking boils down to this:

1. Is there a way to have --user-install use a completely distinct path for each Ruby installation?
2. If there is, would this fully address @eregon's concerns while allowing us to use --user-install by default?

Any questions I asked about how to do so can wait until after we determine if it's helpful. 🙂

[MSP-Greg]

I work with a lot of repos with both Windows and CI issues. Because of that, I have a lot of installed Rubies for local testing, and I also swap patch versions when needed.

Because of that, almost all my gems are installed in --user-install. I don't recall issues with ABI versioning.

The vast majority of my work is with MRI Ruby. I suspect @eregon is often working with non MRI Rubies.

Is there a way to have --user-install use a completely distinct path for each Ruby installation?

Personally, I would like RUBY_PLATFORM as part of the path, as mswin & mingw Ruby builds don't work with each other's gems. But, as to build arguments, that's tricky, maybe a hash or something? Same issue with 32 bit and 64 bit Windows Rubies, but that's less common than in the past (few people use 32 bit).

At some point, --user-install location should be about what almost all 'normal' users of Ruby need. IOW, @eregon's and my needs may just fall outside of that definition.

[eregon]

@duckinator If --user-install could guarantee a distinct path for each Ruby installation, then I would have no objection about the safety of sharing gems. I'm unsure if it's possible though, because so many things (some are rare but they still exist) can influence ABI. --enable-shared, multiple versions of MRI master, probably many of the ./configure flags, the version of the native toolchain used when building Ruby and when installing gems, etc.
Ruby managers achieve this by using the "name" of each Ruby like myruby-2.6.5 which is the directory name in which that Ruby is (e.g., ~/.rubies/myruby-2.6.5) and Ruby managers ensure each Ruby has a unique name.
That's not true in general though, one could have /path1/ruby-2.6.5 and /path2/ruby-2.6.5.

There is still the concern that --user-install gems might not be in PATH, the warning would be useful but it also means users would still need a manual step anyway (so IMHO they might as well add --user-install in their .gemrc), or they need to use a Ruby manager which anyway handles all of that.
Fedora handles that with gem: --user-install --bindir $HOME/bin which is one way, assuming $HOME/bin is in PATH.

My experience with shared gems issues is mostly with MRI and native extensions.
On TruffleRuby it's not an issue as the reported ABI versions is (currently) different for each commit.
That's a way to guarantee not mixing gems (and the fact there are no ./configure flags, and it's always --enable-shared).
So one way to solve this could be to have a more precise ABI version in MRI, or consider more than just RbConfig::CONFIG['ruby_version'] in RubyGems, and that would have to be reflected in the path for gems.
It's bound to be incomplete though, so something based on the Ruby "name"/path seems best IMHO.

[MSP-Greg]

@eregon

There is still the concern that --user-install gems might not be in PATH

I think that is the more common case, and a lot of things will break. I'm glad I'm not involved with jekyll...

I assume you mean the 'bin directory for --user-install gems'...

[eregon]

Thinking again about this I'm convinced "automatically use --user-install if the default gems directory is not writable" is the only reasonable solution:

• It's not disruptive, for the vast majority of Ruby developers which have the default gems directory writable, it works exactly as before. --user-install would break all of these people's workflow (because gem executables would not be in PATH) and needs extensive changes in anything using Ruby to add the gem_home/bin in PATH. I think that's never going to happen for compatibility reasons.
• It addresses the original issue, by providing a way out when the default gems directory is not writable. It requires adding gem_home/bin in PATH for that case, but there is no another way if the whole Ruby is installed in a non-writable location. We can warn if it's not in PATH as a reminder it's needed for that case.
• It still lets distributions customize the behavior if they can make more assumptions, e.g., a single system Ruby on Fedora.
• It's what I ended up when looking at this issue in chruby after years of discussion about this with various people and trying multiple approaches based on my experience with CRuby, JRuby and TruffleRuby.

I think the next step is to create a PR with this approach. I don't think I will have time soon.
So if anyone wants to help and make a PR with this approach, they would be very welcome :)

[postmodern]

Just adding my thoughts to the discussion, since @eregon and myself have been discussing how to set (or not set) GEM_HOME/GEM_PATH in postmodern/chruby#431 for multiple Rubies. I am interested in the thoughts of the RubyGems team. I also want to ensure that future versions of RubyGems won't break chruby, nor will future versions of chruby break RubyGems.

• Please allow overriding GEM_HOME when --user-install (or something similar) is enabled. This would allow Ruby switchers to better control the gem home when --user-install is enabled.
• If --user-install (or something similar) becomes the default, please allow something like --system-install to forcibly install/update the gems in the Ruby's gem root dir. This could be done explicitly via an option flag (ex: --system-install), or automatically by either checking if the Ruby's gem root dir is writable, or if the user is also the owner of the gem root dir, or if the user is root.
• If the --user-install default GEM_HOME is specific to the Ruby's version/ABI/platform/etc, please consider making --env-shebang the default.
• If we use ~/.gem/#{RUBY_ENGINE}/#{RUBY_ABI_VERSION}-#{RUBY_PLATFORM} (or something similar) as the default --user-install GEM_HOME, and there's a slight possibility of an ABI issue with a gem containing C extensions, would it be acceptable to request that the user simply run gem pristin --extensions as a workaround? Are manual workarounds for less common scenarios acceptable for user-experience, or should RubyGems protect against every edge-case no matter how unlikely?

[eregon]

In order,

• I would think GEM_HOME/GEM_PATH will always have precedence, it would be too incompatible to change. (This PR might not implement that but it's a prototype anyway)
• There is --no-user-install.
• --env-shebang is a different although indeed related discussion, is there an issue for that? (there is When installing bin stubs, enable --env-shebang if File.dirname(Gem.ruby) is in $PATH #1049) BTW, --env-shebang is not enough to share executables from all Rubies, because it currently uses the launcher name and not always just #!/usr/bin/env ruby (e.g., #!/usr/bin/env truffleruby on TruffleRuby). Sharing all gems executables also has drawbacks such has which rspec being true if rspec is installed on any ruby but not necessarily the current ruby. Seems a minor issue practice since rbenv has the same effect with stubs (but still, it's less clean, so IMHO only OK for non-writable gem dirs cases). Also, sharing gems executables in a single directory would not work with multiple system Rubies (ruby has to resolve to one of them, and never others).
• My opinion is we should care about edge cases because it breaks even in not-so-rare cases like multiple CRuby dev builds. In any case, regressing there when the gem home is writable seems unacceptable to me (RubyGems would become less safe and increase the number of cryptic errors).

BTW, it seems RubyGems already warns for --user-install if bin/ is not in PATH:

$ gem install --user-install haml
Fetching temple-0.8.2.gem
Fetching tilt-2.0.10.gem
Fetching haml-5.1.2.gem
WARNING:  You don't have /home/eregon/.gem/truffleruby/20.2.0-dev-dc3a61a7/bin in your PATH,
	  gem executables will not run.
Successfully installed temple-0.8.2
Successfully installed tilt-2.0.10
Successfully installed haml-5.1.2
3 gems installed

That's nice, one step is already done. Although it's a bit buried in the output.

[postmodern]

Correct me if I am wrong, but doesn't --user-install rely on Gem.user_dir, which does not appear to respect $GEM_HOME?

$ GEM_HOME="$PWD/test_gems" gem install --verbose rake
...
/home/postmodern/.gem/ruby/gems/rake-13.0.1/.github/workflows/macos.yml
/home/postmodern/.gem/ruby/gems/rake-13.0.1/.github/workflows/ubuntu-rvm.yml
/home/postmodern/.gem/ruby/gems/rake-13.0.1/.github/workflows/ubuntu.yml
/home/postmodern/.gem/ruby/gems/rake-13.0.1/.github/workflows/windows.yml

I'd assume making --user-install the default argument, without changing anything else, would either:

1. Confuse users who are using RVM/rbenv/chruby when they install new gems, as those new gems would be installed into Gem.user_dir instead of $GEM_HOME set by RVM/rbenv/chruby.
2. Force RVM/benv/chruby to assume gems will be installed into whatever path Gem.user_dir typically generates.

I'm interested in hearing everyone's thoughts.

[duckinator]

Thinking again about this I'm convinced "automatically use --user-install if the default gems directory is not writable" is the only reasonable solution:

@eregon If I'm understanding correctly, then yeah I think something like this is probably the solution.

Correct me if I am wrong, but doesn't --user-install rely on Gem.user_dir, which does not appear to respect $GEM_HOME?

@postmodern My understanding is that @eregon's idea can be summarized with the following pseudocode:

if !File.writable?(gem_home)
  default_args += ' --user-home'
end

Which is different than what is done in this PR in that if you don't pass --user-home + GEM_HOME is writable, it will use GEM_HOME.

[hsbt]

#2847 (comment) is the good start point for us.

I update the this pull-request with it.

[eregon]

The PR looks good to me now.

About GEM_HOME/GEM_PATH, I'd think they are already respected, because they influence Gem.paths and obviously, if set, GEM_HOME should be writable:
https://github.com/rubygems/rubygems/blob/4a76ff8029effb6172d8c0327565d9d2b7aa50ff/lib/rubygems/path_support.rb#L26-L34

@postmodern Could you test that holds?

[eregon]

@hsbt Could you add some tests for this PR to make sure it works as intended and keeps working?

[hsbt]

Yes, I try to do that.

But I was suprised the current install_update_defaults_str didn't use with install/update commands. I will investigate them.

[deivid-rodriguez]

I'm also fine with the proposed solution.

Only addition I'd like to propose is to show a warning when this fallback happens, in case the user actually expects to install to the system location and has a permissions issue which she wants to fix.

Maybe something like:

WARNING: The default GEM_HOME (<actual_path>) is not writable, so rubygems is falling back to installing under your home folder. To get rid of this warning permanently either add gem: --user-install to your ~/.gemrc file, or fix your GEM_HOME folder permissions.

Thoughts?

[eregon]

That sounds good, I would just swap both suggestions, as fixing permissions (if possible) is a better fix (for --user-install, one will need to add an extra entry to PATH, and it can mix gems from multiple Rubies while fixed permissions cannot).

[felipec]

Bundler ignores --user-install so this is not the full solution.

This does it:

--- a/lib/rubygems/path_support.rb
+++ b/lib/rubygems/path_support.rb
@@ -23,7 +23,7 @@ class Gem::PathSupport
   # hashtable, or defaults to ENV, the system environment.
   #
   def initialize(env)
-    @home = env["GEM_HOME"] || Gem.default_dir
+    @home = env["GEM_HOME"] || Gem.user_dir
 
     if File::ALT_SEPARATOR
       @home = @home.gsub(File::ALT_SEPARATOR, File::SEPARATOR)

However --no-user-install would need to override this location, or much better: a new --system-install option.

[MichaelCurrin]

@felipec My understanding is this is only about running gem install with or without --user-install flag.

What do you mean Bundler ignores --user-install? Do you mean bundle install --user-install doesn't work (I don't think it will and I don't think it needs to). Or something else like what Bundle does internally to install gems?

[postmodern]

Just to get clarification, is Gem.user_dir meant to be tied to one specific Ruby, or is it intended to be shared among multiple Rubies who happen to have the same RUBY_ENGINE and RbConfig::CONFIG['ruby_version'] (ABI version)? Looking at the current master branch, Gem.user_dir still uses RbConfig::CONFIG['ruby_version']. If we are going to enable this --user-install fallback behavior, there is a good chance multiple Rubies will install gems with executable bin/ scripts into the same Gem.user_dir resulting in bin/ executables with different #!/path/to/ruby. It is not clear whether this is the desired behavior or accidental due to having multiple Rubies installed? Some clarification and guidance on whether Gem.user_dir should or shouldn't be shared by multiple Rubies would be helpful.

[MSP-Greg]

or is it intended to be shared among multiple Rubies who happen to have the same RUBY_ENGINE and RbConfig::CONFIG['ruby_version'] (ABI version)?

It's not a typically situation, but I have issues with --user-install when testing x64-mingw32 & x64-mswin64_140 Ruby master builds (both Windows, different compilers). Fixing the issue would definitely be a breaking change.

Does #!/usr/bin/env ruby solve your issue?

I don't know, I'm only Windows, but that will change soon...

[duckinator]

There were enough changes to the codebase since this was opened that I was able to re-implement this in a much simpler way over at #5327.

Thanks for getting this started, @hsbt!

[duckinator]

Closing in favor of #5327.