Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load binary extensions from separate path #210

Closed
voxik opened this issue Nov 3, 2011 · 24 comments

Comments

7 participants
@voxik
Copy link
Contributor

commented Nov 3, 2011

Hello,

During preparation of Ruby 1.9.3 for F17, I am trying to make sure that Ruby, including RubyGems are going to obey FHS [1]. For RubyGems, it means that they should reside in /usr/share folder, since they are typically platform independent.

Nevertheless, there are also gems with binary extensions. Binary files should be place in /usr/lib{64} and for a long time, there was policy in Fedora to place the binary libraries into ruby_sitearch dir [2]. However, this has the disadvantage, that the files may collide between individual versions of gem.

So in attempt to solve this issue, I've come with first approximation patch [3], which proposes new "exts" folder in the gem directory structure, on the same level as "gems" directory and following the same structure. Once the gem is activated, in the case that the gem has binary extensions, the binary extension is pushed into the $LOAD_PATH as well as the original gem files location.

I said first approximation, because the ultimate goal for me is to load the platform independent part of gems from /usr/share folder where as the binary extensions should be loaded from /usr/lib{64}. Please, do you think that this approach is feasible? Would you accept such patch? Or do you have some other suggestions or plans in such direction?

[1] http://www.pathname.com/fhs/pub/fhs-2.3.html
[2] https://fedoraproject.org/wiki/Packaging:Ruby#Ruby_packages_with_binary_content.2Fshared_libraries
[3] https://gist.github.com/1336824

@voxik

This comment has been minimized.

Copy link
Contributor Author

commented Nov 3, 2011

Btw the patch applies cleanly to RubyGems v1.8.11

@raggi

This comment has been minimized.

Copy link
Contributor

commented Nov 3, 2011

Presumably you are sending equivalent patches to ruby-core for stdlib stuff, and for setup.rb's?

@zenspider

This comment has been minimized.

Copy link
Contributor

commented Nov 3, 2011

Please see #206 for platform specific patches. Other platforms don't care about FHS and it shouldn't be the default.

@ghost ghost assigned evanphx Nov 3, 2011

@voxik

This comment has been minimized.

Copy link
Contributor Author

commented Nov 4, 2011

@raggi, yes, I sent a few patches to ruby-core to allow to move around some ruby folders:

http://redmine.ruby-lang.org/issues/5281
http://redmine.ruby-lang.org/issues/5231

@zenspider
I am well aware about #206 and I still believe this feature should be considered. Yes, we can use operating_system.rb, yes, we can patch RubyGems, but if we don't have to, it would be nice. That is why I submitted this patch and I expected more discussion than "we don't care".

@zenspider

This comment has been minimized.

Copy link
Contributor

commented Nov 4, 2011

Did I just say "we don't care"? Oh look! No... No, I didn't... Please don't put words in my mouth. Also, I left this ticket open, so it is still open for discussion.

What I suggested was that the changes you want specific to your operating system should go into operating_system.rb. Other operating systems should not have these changes as they're simply not applicable. If something needs to be refactored to accommodate your changes better, we'll be more than happy to consider it.

@voxik

This comment has been minimized.

Copy link
Contributor Author

commented Nov 8, 2011

My patch consist from 3 modifications:

  1. When the RubyGems are trying to find if the gem has the file to require, there is looked not only the default path but also some path dedicated binary extensions path, but only for gems which contains binary extensions.
  2. There is added the dedicated binary extensions path to the search path, but only for gems with binary extensions.
  3. Path setup. This needs to be overridden in operating_system.rb for Fedora anyway.

So now I am wondering what are your thoughts. It could be switched off by several ways. Here are some which comes to my mind:
a) Do not define the ext dir paths methods at default. Only if they are defined in operating_system, then the extension will be used
b) Similar to above, exept there will be default implementation of the ext dir methods, but it will return just nil. So unless it will be redefined in operating_system.rb, the extensions directories will not kick in.
c) Use some environment switch, or possibly setting in operating_system.rb, which would globally allow some FHS extensions.
d) Some other?

Could you please elaborate a bit about which direction would be acceptable, or where should I focus my effort?

@bkabrda

This comment has been minimized.

Copy link

commented Nov 11, 2011

I think that the voxik's proposed changes should be considered as they target not only Fedora, but all FHS compliant systems. Would it be possible to consider which of the variants that he proposed can be implemented? Personally, I like b) and c), as they are systematic solutions and could be left turned off.
What do you think, @zenspider?

@voxik

This comment has been minimized.

Copy link
Contributor Author

commented Dec 13, 2011

Hi,

I have worked on the issue further on and you can find the result in my fork of RubyGems [1]. The corresponding operating_system.rb, which I'd like to use for Fedora is here [2]. Of course RubyGems works the same way as they always worked without the customized operating_system.rb, so no other platform should be impacted.

[1] https://github.com/voxik/rubygems/tree/fhs-binary-extensions
[2] https://gist.github.com/1472698

@lutter

This comment has been minimized.

Copy link

commented Feb 10, 2012

Are there any opinions on voxik's patches ? Getting this straightened out would be very helpful for the state of Ruby not just in Fedora, but in lots of Linux distros.

@luislavena

This comment has been minimized.

Copy link
Member

commented Apr 20, 2012

@voxik I see a problem with this, and is the case you're assuming extensions (from gemspec) will not be empty.

The change will only affect gems installed with ruby platform that compiled extensions during installation, since those are the only ones that have extensions in their specs.

This will affect gems that pre-compiled extensions and extensions in the gemspec is empty.

I'm working on a successor of rake-compiler that will leverage packaging of binary gems from outside the gem, so modifications to Gem::Installer will affect it.

I do think platform as namespace of the compiled binaries is a good idea, but I'm not sure about the approach to it.

@voxik

This comment has been minimized.

Copy link
Contributor Author

commented Apr 21, 2012

@luislavena Not sure what you are referring to. Yes, of course this affect only gems with extensions compiled during installation. How it should affect the gems with pre-compiled extension?

@luislavena

This comment has been minimized.

Copy link
Member

commented Apr 21, 2012

How it should affect the gems with pre-compiled extension?

Because:

I'm working on a successor of rake-compiler that will leverage packaging of binary gems from outside the gem, so modifications to Gem::Installer will affect it.

Since the compiled extensions will be placed on a platform-namespaced directory inside lib, it will no longer found in the path. The modification you're suggesting will not add this platform-namespaced directory to the search path so it will not work.

@voxik

This comment has been minimized.

Copy link
Contributor Author

commented Apr 21, 2012

Luis, could you please show me some example directory structure, code, etc to let me understand better?

BTW I see no reason why it should not work. The patch do not modify any current functionality of RubyGems unless you enable it in operating_system.rb [1]. If the binaries on some platform are placed into some other folder, it might be adjusted.

[1] https://github.com/voxik/ruby.spec/blob/master/operating_system.rb#L65-68

@bkabrda

This comment has been minimized.

Copy link

commented Jun 15, 2012

@luislavena tested it, works fine. Are there any other concerns about voxik's patches?

@bkabrda

This comment has been minimized.

Copy link

commented Jun 19, 2012

To clarify my last comment:
There is no reason why the patches shouldn't work, as they only add some more paths. And even that, they do only if enabled by operating_system.rb. Therefore everything will work fine, no matter what directories you add inside lib - if they are on load path now, they will also be present after applying the patches.

@bkabrda

This comment has been minimized.

Copy link

commented Jul 11, 2012

@luislavena @zenspider Guys, any reactions on this one? Would you still consider the proposed patch?
Thanks!

@zenspider

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2012

I'm out of this discussion. I leave it to @evanphx.

@raggi

This comment has been minimized.

Copy link
Contributor

commented Jul 12, 2012

The outstanding backward compatibility issue I see (although I don't feel it should be necessarily supported), is when people are modifying the load path themselves in order to make subtrees of ext loadable.

There is a further problem that extensions are not only built by mkmf, in the case of the alternative methods, there is a lot of potential for failure of this implementation. I do feel that it might be a good idea to consider removal of non-extconf extensions from RG 2.0, but that might be too soon, and deprecation may be preferred. Alternatively adding a specific guide for how and where the alternative extension systems should put files, with documentation explaining where outputs will be moved after completion.

@bkabrda

This comment has been minimized.

Copy link

commented Jul 13, 2012

Hmm, I don't see any potential failure. If alternative methods of extensions building don't move them anywhere, the extensions remain in the old load path. This is precisely the beauty of this approach. It doesn't remove any load paths, only adds them for extension gems. Worst case, the extension will stay in the "non-ext-directory" (the one that is currently added to load path) and everything will still work.

@bkabrda

This comment has been minimized.

Copy link

commented Jul 17, 2012

BTW I hope we're both referring to [1], which doesn't depend on the customized operating_system.rb - the patch from initial comment does.

[1] https://github.com/voxik/rubygems/tree/fhs-binary-extensions

@bkabrda

This comment has been minimized.

Copy link

commented Aug 21, 2012

Ping, any more thoughts on this one? I really think that there is no problem in this approach. If there is, I'd like to see a use case, that would fail in loading the gem's extensions, so I could work on the patch further (but I don't really think that there is such a case).

@bkabrda

This comment has been minimized.

Copy link

commented Nov 2, 2012

@raggi @evanphx: guys, any further thoughts on this one? I really see no problems in this approach. If there is a problem, could you please specify a usecase, that would break, so that the proposed patches can be fixed/updated? I'd really like to see this in rubygems, so I'm wondering what's stopping it.
Thanks!

@voxik

This comment has been minimized.

Copy link
Contributor Author

commented Dec 5, 2012

I updated my branch [1] against master.

This patch would be useful not only to Fedora, but also Rubinius might benefit from it. It allows to have installed one copy of gem for 1.8 and 1.9 modes while its binary extensions would be in separated folder.

[1] https://github.com/voxik/rubygems/tree/fhs-binary-extensions

voxik added a commit to voxik/rubygems that referenced this issue Sep 2, 2013

Add full_require_paths.
Add convenient method, which returns expanded version of
require_paths. This removes code duplication and may helps rubygems#210 a bit.
@voxik

This comment has been minimized.

Copy link
Contributor Author

commented Dec 13, 2013

Since #744 was merged, it seem that all pieces of this puzzle are in place now. Thanks to everybody who participated in this thread.

@luislavena BTW are your concerns from #210 (comment) about precompiled gems addressed?

@voxik voxik closed this Dec 13, 2013

nobu pushed a commit to nobu/rubygems that referenced this issue Jan 12, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.