support for GEM_HOME_SHARED #596

Closed
wants to merge 3 commits into
from

7 participants

@mpapis

would it be possible to support second GEM_HOME like GEM_HOME_SHARED which would be used for gems that do not use in specification:

this way it would be possible to use single GEM_HOME_SHARED for all rubies and only manipulate GEM_HOME when changing rubies for gems with extensions or platform specific

@zimbatm

Even better if rubygems is able to lazily-compile the extension and put it in a "#{RUBY_ABI}-#{ARCH}-#{KERNEL}" prefix then all the gems can be shared.

@postmodern

You could accomplish this currently by just manipulating GEM_PATH accordingly.

@postmodern

I also like the idea of grouping the compiled extension libraries by platform/arch, allowing them to be recompiled for other Rubies/Systems.

@mpapis

@postmodern the idea is to have two active GEM_PATHs for gem installation - one for gems that do not have c-extensions or depend on platform - second for those platform specific; so the first gems will be installed to location shared between multiple ruby versions, and the second ones get installed into ruby/version specific location

@postmodern

@mpapis I agree with @zimbatm. We shouldn't be using ENV variables to group gems, there should be an internal structure within the gem dir, that properly organizes the gems. We also should be mindful of ENV variable pollution.

@mpapis

how this would work for http://docs.rubygems.org/read/chapter/20#required_ruby_version

spec.required_ruby_version = '>= 1.6.8'

also #{kernel} is not good check, how would it handle libc changes?

@postmodern

@mpapis ideally the extensions would be compiled against specific ABIs of the libraries (ex: /usr/lib/libFLAC.so.8).

As for required_ruby_version, I feel that it should be deprecated in favour of self.ruby_version_family = '1.9'; or something similarly named.

@mpapis

after some digging:

puts Gem::Specification._all.group_by{|s| s.required_ruby_version.requirements }.
  map{|k,v| "#{k*" "}: #{v.map{|s|s.name}.uniq.count}"}.sort
>= 0: 55
>= 1.8.2: 1
>= 1.8.6: 2
>= 1.8.7: 3
>= 1.9.2: 2
>= 1.9.3: 1

this option would play nice with --user-install paths:

$ rvm 1.8.7,1.9.3,2.0.0,jruby-1.7.3,rbx-head,rbx-head-d19 --verbose do \
   ruby -rubygems -e 'puts Gem.user_dir'
ruby-1.8.7-p374: ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux] 
/home/mpapis/.gem/ruby/1.8
ruby-1.9.3-p448: ruby 1.9.3p448 (2013-06-27 revision 41675) [x86_64-linux] 
/home/mpapis/.gem/ruby/1.9.1
ruby-2.0.0-p247: ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-linux] 
/home/mpapis/.gem/ruby/2.0.0
jruby-1.7.3: jruby 1.7.3 (1.9.3p385) 2013-02-21 dac429b on OpenJDK 64-Bit Server VM 1.7.0_21-b02 [linux-amd64] 
/home/mpapis/.gem/jruby/1.9
rbx-head: rubinius 2.0.0.rc1 (1.8.7 bed7e913 yyyy-mm-dd JI) [x86_64-unknown-linux-gnu] 
/home/mpapis/.gem/rbx/1.8
rbx-head-d19: rubinius 2.0.0.rc1 (1.9.3 6f2a1d90 yyyy-mm-dd JI) [x86_64-unknown-linux-gnu] 
/home/mpapis/.gem/rbx/1.9
@mpapis

changes:

  • behavior of --user-install - detects if gems can_be_shared? and use Gem.shared_user_dir
  • uses "#{Gem.shared_user_dir}/cache" over "#{Gem.user_dir}/cache" - so the cache is shared for --user-install

WIP - still missing:

  • tests
  • docs?
  • I would like to add GEM_HOME_SHARED for cases where GEM_HOME was used, so gems can be shared also when used with variables.
@zimbatm

Nitpick: I think the pattern here would ask the method to be called "cache_dir"

there is bindir, datadir and there is spec_cache_dir - I just used the pattern that was the closest - bindir

RubyGems member

.cachedir is fine.

@zimbatm

I actually had the idea of a shared +cache_dir+ for a while now but never got around to finish it. See zimbatm/rubygems@f426069 . It is depending on a GEM_CACHE environment variable with the user_dir/'cache' as a default.

It would be better if gems could be shared in their installed form with a lazy compile hook but it requires more effort. We need to change the compilation process to direct the compiled output to a ABI+ARCH+... specific folder (plus introduce the hook). Right now I believe most of the gems just put the .so in their lib/ folder.

@evanphx
RubyGems member

Can we rewind and state the problem to be solved? Is it that you'd like a path containing gems to be shared across ruby versions?

@mpapis

assumption:

Gem.shareddir=ENV["GEM_HOME_SHARED"] if ENV["GEM_HOME"]

so shareddir has no effect outside of GEM_HOME rubies ... not sure if t is good or bad

@evanphx evanphx commented on an outdated diff Jul 15, 2013
lib/rubygems/uninstaller.rb
@@ -68,6 +81,14 @@ def initialize(gem, options = {})
@user_install = options[:user_install] unless options[:install_dir]
end
+ def gem_in_uninstallable_path(base_dir)
@evanphx
RubyGems member
evanphx added a line comment Jul 15, 2013

Restructure this method please!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@evanphx evanphx commented on an outdated diff Jul 15, 2013
lib/rubygems/uninstaller.rb
@@ -275,6 +288,27 @@ def remove(spec)
end
##
+ # Confirm spec is one of paths that it can be uninstalled from
+
+ def paths_ok_or_raise(spec)
+ unless (
@evanphx
RubyGems member
evanphx added a line comment Jul 15, 2013

Please restructure this conditional!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@drbrain
RubyGems member

this way it would be possible to use single GEM_HOME_SHARED for all rubies and only manipulate GEM_HOME when changing rubies for gems with extensions or platform specific

Why not have RubyGems just Do the Right Thing™ to avoid the confusion of an extra environment variable. I don't think "SHARED" is clear enough by itself. I also want to avoid answering questions about what this extra environment variable configures.

I'm guessing that this sharing is meant to be between ruby versions compiled on the same machine. This suggests changing the way gems are installed (a special directory, as @zimbatm described) is the way to go. This makes all the details transparent to the user.

@mpapis

Do the Right Thing™:
1. it's implemented here for :user_install and for given GEM_HOME+GEM_HOME_SHARD (see: note)
2. the problem is for default location when you install all rubies to one place (--prefix) with different --program-suffix/--program-prefix like: jruby / ruby19 ruby20, then selecting gems path is different then for :user_install like the jruby default is to host gems in gems/shared - which would conflict with this feature.

note: unless you want to follow @zimbatm proposal with sharing ruby code to only change location where the c-extensions are compiled (platform paths) and do lazy compile of extensions during require if they do not exist for given platform, but this brings up problem with platform dependent code where it's already shipped with the gem like for bouncy-castle-java which is built with the jars embedded => https://github.com/jruby/jruby/blob/master/gems/bouncy-castle-java/Rakefile#L12-L13 - because there is no need to compile the java "native" code on the client side.

This pull request acts like a safe way to start doing that which will work with already installed gems without breaking compatibility or doing big changes.

@mpapis

continuing problems for 2., what if rubies are installed in different prefixes, what is the common denominator for them /var/ruby/shared?

@drbrain
RubyGems member

This pull request acts like a safe way to start doing that which will work with already installed gems without breaking compatibility or doing big changes.

I would like a solution to this problem that doesn't require configuration changes by third parties (rvm, jruby, ruby).

I'm not worried so much about big changes, I would rather transition from a world where each ruby needs a separate GEM_PATH to a world where they can all live together in one go than have to support (through bug fixes) multiple steps.

I'm not particularly concerned about compatibility. We should be able to hide them behind Gem::Specification. If the layout of the repository changes (such as installing extensions outside "#{Gem.dir}/gems", see below) can be repaired with gem pristine if a user needs to revert their RubyGems (unlikely, as the API is supposed to be stable for 2.x).

We can already support multiple platforms in a single repository leaving required_ruby_version and extensions.

If the resolver and activation know about required_ruby_version this leaves only the problem of extensions.

For extensions, we could install them in "#{Gem.dir}/extensions/#{spec.full_name}/#{Gem::Platform.local}", include this directory in Gem::Specification#require_paths and build it if it does not exist.

Will this idea work for rvm without needing changes to rvm?

@mpapis

yes the target here is to support one GEM_HOME for all rubies

detect the platform quite specifically like fedora-x86_64-static:
The problem with current platform is it is not specific enough x86_64-linux - this can go wrong in many ways especially if the GEM_HOME would be shared between servers, also the static is important as it happened already multiplr times with bundle --path (equivalent of GEM_HOME=./vedor/gems) that the gems build with dynamically linked ruby tried to load wrong ruby when used in statically linked ruby.

In place compilation on require:
The problem is that right now extensions are build during the gem installation, with the new approach installing a gem for ruby-1.9.3 would compile the code only for this ruby, using the same gem from ruby-2.0.0 would require compiling it again - where the gem is already installed - so it will have to be compiled when the gem is required - this produces new kind of runtime exception - GemExtensionCompilationFailed.

as for compatibility with rvm - it would require making some changes - but should not be that hard, the simplest thing would be if an supper variable was introduced like GEM_HOME_SHARED which was not supported by early versions so it's easier to maintain old rubies with old rubygems and new rubies with new rubygems at the same time.

@drbrain drbrain added a commit that referenced this pull request Sep 19, 2013
@drbrain drbrain Extract find_possible out of resolve_for
find_possible is expected to grow new features as support for #596 is
added.
bb23639
@drbrain
RubyGems member

I tried to implement the required_ruby_version check in the resolver, but it is not yet possible as the index does not contain this information.

@indirect is working on that, and when he has a new index it will be very easy to add.

@mpapis

there is other case when gem is build for generic and specific platform, like it is in case of json gem - where two versions are available - generic(ruby) and java.

there might be few solutions we discussed with @drbrain at #rubygems:

  • install all gem variants for different platforms and compile only extensions for current platform (or in minimal version add a flag for the extra platforms so the gem will not be listed when using the extra platform)
  • install generic(ruby) gem to /platform subdir when more then one platform available

this problem will be also relevant for windows users (x86-mingw32)

@drbrain drbrain added a commit that referenced this pull request Oct 4, 2013
@drbrain drbrain Add extensions directory to GEM_HOME
The extensions directory will hold built extensions for installed gems
so a gem directory can be shared across ruby versions and platforms.

See #596
09f541c
@drbrain drbrain added a commit that referenced this pull request Oct 4, 2013
@drbrain drbrain Add Gem.ruby_api_version
Extensions must be recompiled for each Ruby API version so we need an
accessor for it.

See #596
f7965a2
@drbrain drbrain added a commit that referenced this pull request Oct 4, 2013
@drbrain drbrain Add Specification#extension_install_dir
This is the platform and ruby-specific directory where extensions will
be installed when #596 is fixed.  I think I need to detect
statically-linked ruby, too, but I'll leave that for a future commit
because I don't have a statically-linked ruby to test against it yet.
9485ca2
@drbrain drbrain added a commit that referenced this pull request Oct 7, 2013
@drbrain drbrain Mention #596 in History
This does not mean work for this issue is complete, at the present time
the following items are missing:

Behavior when the destination directory is not writable (exception or
build into ~/.gem).

Behavior for static vs shared ruby.
3a924d3
@drbrain drbrain added a commit that referenced this pull request Oct 7, 2013
@drbrain drbrain Merge branch 'multiplatform_repository'
* multiplatform_repository: (25 commits)
  Mention #596 in History
  Clean before building extensions
  Always write gem_make.out
  Move extension building to file existence check
  Initialize stub specification in setup
  Use stub_with_extension in all tests
  Use full_require_paths in contains_requirable_file?
  Move require_paths up to BasicSpecification
  Include extensions in stub specification data
  Fix test names
  Fix :nodoc: tags
  Build missing extensions at gem activation time
  Added Specification#gem_build_complete_path
  Update references to ExtensionBuildError
  Touch .gem.build_complete upon extension install
  Rename extension build error constant
  Allow Gem::Ext::Builder to accept only the spec
  Fix comment for Gem::Ext::Builder.new
  Complete TODO for using Specification#original_name
  Install extensions into a separate directory
  ...
cb5c07d
@drbrain drbrain added a commit that referenced this pull request Oct 7, 2013
@drbrain drbrain Separate extensions for shared and static ruby
Ruby built with ENABLE_SHARED cannot share extensions with ruby built
without this option, you get an immediate segmentation fault on load.

Now the extension directories are separate per ruby API version.
Static ruby now has a "-static" appended.

This addresses one of the TODOs in @3a924d3, see also #596
f8fcb99
@drbrain drbrain added a commit that referenced this pull request Oct 7, 2013
@drbrain drbrain Do not clean for cmake
Makefiles created with cmake do not seem to have a general "clean"
target.  This may mean that this commit for #596 breaks cmake support,
but I don't know of a cmake project to test it with.
010ae97
@drbrain
RubyGems member

The only remaining question I have is one of behavior:

What should RubyGems do if the gem repository is not writable but it needs to build an exception?

Possible solutions:

  1. Raise an exception. For good user behavior, this would need to be handled by the gem activation code as a gem from a different directory in GEM_PATH may be usable or installable. This behavior means users who sudo gem install will have consistent behavior, but I don't know if it is expected (as this feature is new).
  2. Install the extension somewhere under ~/.gem. The user should always be able to write to their home directory. This means that RubyGems will continue to Just Work™. This involves more code, but should have a better user experience.
@postmodern

@drbrain I like the idea of RubyGems selecting the first writable gem dir as it's primary directory for writable actions. This would allow Debian's RubyGems to install into ~/.gem/ instead of /var/lib/gem/... by default.

@indirect
RubyGems member
@mpapis

to make use of this feature users would have use GEM_HOME - I assume rarely paths not writable by user are used for it.

a few more questions:

  1. how it will work with --user-install as there is already a structure in ~/.gem
  2. what if I have non-extension gem (A) that depends on extension gem (B) and gem20 install A; gem21 install A - gem install ... --dependency-extensions?
@drbrain
RubyGems member

@postmodern writing to an arbitrary directory in GEM_PATH makes it difficult to uninstall properly, especially if the user were to alter GEM_PATH. It will be easiest for me to maintain and understand if the extension is alongside the gem sources or in one other well-defined location.

I think I will implement solution 1. How often do users set GEM_HOME to a directory they need sudo for write? For vanilla ruby installs, there will be no sharing, so such users will not encounter this exception.

@postmodern

@drbrain good point. Then I would fall back to Gem.user_dir/$GEM_HOME.

@drbrain drbrain added a commit that referenced this pull request Oct 9, 2013
@drbrain drbrain Do not attempt extension build for legacy repos
For legacy repositories, RubyGems would try to build extensions that
were already built.  This would result in confusing failures to load
gems with confusing messages.

Now RubyGems does not attempt to build extensions when the "extensions"
directory does not exist for the local repository and the repository
root is not writable.

See #596
1df8c2a
@drbrain drbrain added a commit that referenced this pull request Oct 12, 2013
@drbrain drbrain Compile missing extensions during installation
After #596 if a user installs a gem into a shared repository any missing
extensions in dependency tree should be installed at the same time.

This should reduce the amount of compiling required during activation,
and may make compiling during activation unnecessary (but I will need to
check it.
93d8ec7
@voxik

@postmodern

This would allow Debian's RubyGems to install into ~/.gem/ instead of /var/lib/gem/... by default.

If Debian wanted they could do something like we do in Fedora. gem install is installing gems into ~/.gem by default, except root, who can manage system wide gems in /usr/local. It is not that hard. That is why there exists something like operating_system.rb, where you can rewrite Gem.default_{dir,path,bindir}.

@drbrain

What should RubyGems do if the gem repository is not writable but it needs to build an exception?

Only the solution 1 works for Fedora. We definitely don't want RubyGems to mess with packaged gems. This is responsibility of distribution and its packaging system.

Also, building something into someone's home directory doesn't make sense. That would lead to some exceptional behavior and hard to track issues.

@voxik

One general remark to this issue. In Fedora, we definitely support the idea of sharing gems between different implementations. However, it has to be definitely in the "Do the Right Thing™" way. I.e. for us, the only reasonable solution is to share everything by default. So I am very grateful for the recent development.

@drbrain
RubyGems member

@voxik thanks, I'll implement solution 1 then, I was already leaning toward it as it seems easier and more straightforward to implement.

@drbrain
RubyGems member

This is implemented in RubyGems 2.2.0.

@drbrain drbrain closed this Nov 26, 2013
@mpapis mpapis deleted the mpapis:features/gem_home_shared branch Nov 26, 2013
@mpapis

thank you!

@jacknagel jacknagel referenced this pull request in tpope/vim-bundler Apr 18, 2014
Closed

Support for rubygems 2.2 directory structure #36

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment