Skip to content

Loading…

Static and shared libraries for rubinius #1043

Closed
chickenkiller opened this Issue · 19 comments

6 participants

@chickenkiller

Right, so I reopen the same topic as issue #809 after getting my feet wet about the topic.

The commit chickenkiller/rubinius@00e764a attempts to make static the static library useable, and to improve the overall process.

I'd like to get as much feedback as possible from interested people because I have no way to test this on other platforms than Linux. @jameskilton, can you confirm I've not broken the darwin platform? It's also possible that your workaround for ffi_environ is not necessary anymore, but I'm not 100% sure.

The gist https://gist.github.com/911589 should still work without modification.

Thanks everyone for your feedback, I know there's still a lot of room for improvement.

Cheers
Lionel

@chickenkiller

Hi, here's a short status.

First, some resources:

  • The librbx branch (based on master) can be found here.
  • The comparison against the official master branch is visible here.

Then, what I did: I made a few other things work, so right now I am able to:

  • compile a working shared library librubinius.so
  • compile a working static library librubinius-static.a
  • use correct flags in rbconfig.rb so that it's very easy to compile and link an executable using one of those (keep in mind that when linking against the static library, all other deps must be given, particularly LLVM libraries. rbconfig.rb takes care of this now)
  • compile all extensions correctly (they mostly use the same command-lines as before, but I met issues reproducing these command lines correctly with mkmf.rb)

In short... it works. :+1:

The same example as usual works without modification.

What remains to do:

  • code cleanup (I duplicated some functionality from daedalus/rakelib to the configure script)
  • test on other platforms (I need you all! I only have linux available...)

So thanks everyone for your feedback.

@bakkdoor
Rubinius member

+1

@jasonroelofs

It unfortunately doesn't build on OS X 10.6 as it's gcc doesn't support -soname in

gcc -shared -Wl,-soname,librubinius.so -install_name ...

Looking at this commit: chickenkiller/rubinius@5bb050a it looks like what I had probably wasn't doing anything at all (W1 instead of Wl, oops), not sure what the right way of doing this is though. Getting past that, the static library build dies with a bunch of

1: AR vm/librubinius-static.a
ar: .tmp_objs/libffi/__.SYMDEF: No such file or directory
ar: SORTED: No such file or directory
ar: .tmp_objs/libgdtoa/__.SYMDEF: No such file or directory
ar: SORTED: No such file or directory
ar: .tmp_objs/libtommath/__.SYMDEF: No such file or directory
ar: SORTED: No such file or directory
ar: .tmp_objs/libLLVMMCDisassembler/__.SYMDEF: No such file or directory
ar: SORTED: No such file or directory
ar: .tmp_objs/libLLVMLinker/__.SYMDEF: No such file or directory
ar: SORTED: No such file or directory
ar: .tmp_objs/libLLVMX86Disassembler/__.SYMDEF: No such file or directory
ar: SORTED: No such file or directory
@chickenkiller

Hi, thanks for testing!
I think I corrected the first point, give the new librbx head a try and tell me...
For the second problem, I suspect the ar executable on MacOS of not having the same command-line options. I tend to stick on GNU versions so that does not surprise me much. I'll try to dig into some MacOS docs.

@chickenkiller

I may have corrected the second point as well now. Please check and tell me.

@chickenkiller

Anyone checked the latest version of my librbx branch? I'd really need some feedback from other users !

Thanks in advance

@nasser

For static linking on OSX, I use libtool instead of ar like this

libtool -static -o librbx.a *.o

I checked out your branch, but don't have time to write any code at the moment. I'll try this out as soon as I can, though. Where should I look to modify the linking code?

I am developing a Processing-like language based on Ruby called Zajal, and the limitations of MRI have become too much to deal with. I'm looking into moving to Rubinius and this is exactly what I need.

@chickenkiller

Hi nasser,

Thanks for the pointer to libtool, but I think I could solve thear problem, though I'm not sure the resulting library is working correctly. It would be nice if you could try to link a static program and tell me if that works out-of-the-box (I manually remove some __.SYMDEF archive members actually created by libtool).

From the page you gave, I seem to understand that libtool is meant as an smart ar replacement, so I guess we could benefit from it. I also learnt thanks to you that Apple provides a libtool utility which is by no means the same kind of utility as GNU libtool which I'd like to avoid as much as possible.

About the code, around here you have a static and a shared method that do the stuff. I apologize in advance for the dirty hack in the static method... Feel free and be welcome to change that, I'd be glad to know a better solution (maybe libtool indeed).

For testing, you can try to clone https://gist.github.com/911589 and do rake build_static once the rake install step is successful on librbx branch.

Cheers!

@nasser

without modifying anything, i did

$ ./configure
$ rake

it gets to the point where it seems to be compiling C extensions and fails like this

(in /Users/nasser/Workspace/librbx/rubinius/lib/ext/melbourne)
Building Melbourne for Rubinius
CXX bstrlib.c
CXX grammar.cpp
CXX melbourne.cpp
CXX node_types.cpp
CXX quark.cpp
CXX symbols.cpp
CXX var_table.cpp
CXX visitor.cpp
LDSHARED rbx/melbourne.bundle
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -S rake  -r /Users/nasser/Workspace/librbx/rubinius/config.rb -r /Users/nasser/Workspace/librbx/rubinius/rakelib/ext_helper.rb -r /Users/nasser/Workspace/librbx/rubinius/rakelib/dependency_grapher.rb build
(in /Users/nasser/Workspace/librbx/rubinius/lib/ext/nkf)
CC nkf.c
LDSHARED nkf.bundle
Building openssl
/Users/nasser/Workspace/librbx/rubinius/bin/rbx-build extconf.rb 
=== OpenSSL for Ruby configurator ===
=== Checking for system dependent stuff... ===
checking for t_open() in -lnsl... An exception occurred running extconf.rb
    The complier failed to generate an executable file.
You have to install development tools first.
 (RuntimeError)

Backtrace:
                    Object#try_do at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:317
                 Object#try_link0 at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:366
                  Object#try_link at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:370
                  Object#try_func at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:478
       { } in Object#have_library at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:673
       { } in Object#checking_for at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:618
          { } in Logging.postpone at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:240
                     Logging.open at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:213
          { } in Logging.postpone at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:240
                     Logging.open at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:213
                 Logging.postpone at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:236
              Object#checking_for at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:617
              Object#have_library at /Users/nasser/Workspace/librbx/rubinius/lib
                                     /mkmf.rb:668
                 Object#__script__ at extconf.rb:39
  Rubinius::CodeLoader#load_script at kernel/delta/codeloader.rb:65
  Rubinius::CodeLoader.load_script at kernel/delta/codeloader.rb:90
           Rubinius::Loader#script at kernel/loader.rb:618
             Rubinius::Loader#main at kernel/loader.rb:762
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/Users/nasser/Workspace/librbx/rubinius/bin/rbx
        --with-openssl-dir
        --without-openssl-dir
        --with-openssl-include
        --without-openssl-include=${openssl-dir}/include
        --with-openssl-lib
        --without-openssl-lib=${openssl-dir}/lib
        --with-kerberos-dir
        --without-kerberos-dir
        --with-kerberos-include
        --without-kerberos-include=${kerberos-dir}/include
        --with-kerberos-lib
        --without-kerberos-lib=${kerberos-dir}/lib
        --with-debug
        --without-debug
        --enable-debug
        --disable-debug
        --with-nsllib
        --without-nsllib
rake aborted!
Command failed with status (1): [/Users/nasser/Workspace/librbx/rubinius/bi...]

(See full trace by running task with --trace)

it seems like openssl is failing to compile. it complains about missing development tools, but my clone of rubinius compiles just fine.

i'm looking into it. let me know if you have any ideas.

@chickenkiller

ah yes, reminds me something. I already had issues with openssl, because it appends to its own compiling command-lines some options that we expect once librbx is ready, to link against it. Actually we don't expect these flags for an extension, but it seems that's what mkmf.rb does.
So there's probably somewhere in linkflags a syntax which is not supported by your toolchain. Could you put the mkmf.log in a gist to start with?

@nasser

it seems to be coming up blank. am i looking at the right file?

$ pwd
/Users/nasser/Workspace/librbx/rubinius
$ cat lib/ext/openssl/mkmf.log 
$ 
@chickenkiller

Yes, I also happened to have it blank sometimes. I remember having added dirty traces in lib/mkmf.rb to get some results. Their logging stuff doesn't seem to always work.
Can you try to apply the patch here: https://gist.github.com/1118278
the following command may help, execute in rubinius root folder:

wget --no-check-certificate -O - https://raw.github.com/gist/1118278/0b1c9035c4593972db38f20fa28c57e28d89fb59/gistfile1.diff | patch -p1

then a new rake should generate lib/ext/openssl/mymkmf.log.

Thanks in advance,
Lionel

@nasser

not sure how useful this is. i noticed the build process, although braking on openssl, generated a librubinius-static.a and librubinius.so in the vm folder, so i tried to compile it anyway.

https://gist.github.com/1118833

@chickenkiller

about https://gist.github.com/1118524: you should try to execute the commandline in lib/ext/openssl and see what errors the toolchain gives. Maybe then, trying to remove/change some options until the commandline works... Can you give it a try on your MacOS ?

@chickenkiller

about https://gist.github.com/1118833: you're missing plenty of libraries (and options) in your command-line. The fact is that rbconfig.rb helps you construct the command-line, like shown in the gist made by @jameskilton (https://gist.github.com/911589).

I suggest that you do the following:

$ bin/rbx -rrbconfig -e 'c=RbConfig::CONFIG; puts %|-L#{c["libdir"]} -I#{c["includedir"]} -I#{c["includedir"]}/rubinius #{c["LIBRUBYARG_STATIC"]}|'

This will show you the options to put on you command-line.

@intractable

Hi everyone,

I recently resurrected this stuff, as I'm preparing to work on the embedding API a bit; my librubinius branch contains the changes I made to get things working against the current rubinius master.

Almost all of the code was initally merged in from nasser's librbx_osx_fix branch, which seemed to best capture the work done by others previously. I mostly just made minor tweaks.

Note that header files are not currently marshaled to the installation include directory; I figured that hand-rolling final executables (with the appropriate -I switches and the like) for the time being is probably better than exposing too much of the Rubinius internals. I figure that this aspect will get itself sorted out as I start to work more on the embedding API, but I wasn't sure if folks preferred that the headers are wholesale copied for the time being -- feel free to let me know if I ought to do something different here.

I've only tested this on OS X 10.7.3; if anyone feel like testing this on other platforms, that'd be fantastic!

My slightly-modified test program, along with a test script for linking, can be found here. Let me know if there's anything else I can do to help get this up to master.

@nasser

Nice! This is really exciting! I've resumed work on this project full time, so I'll be making use of these updates immediately.

There's a budding conversation around embedding Rubinius over at my pull request. You should join in. I'm also going to contact the core team to talk to them about working more seriously on the API. It seems like a lot of it is in place and just needs a little attention to make it usable.

@brixen brixen added a commit that referenced this issue
@brixen brixen Updated rubygems to 2.4.5.
RubyGems 2.4.5 installed

=== 2.4.5 / 2014-12-03

Bug fixes:

* Improved speed of requiring gems.  (Around 25% for a 60 gem test).  Pull
  request #1060 by unak.
* RubyGems no longer attempts to look up gems remotely with the --local flag.
  Pull request #1084 by Jeremy Evans.
* Executable stubs use the correct gem version when RUBYGEMS_GEMDEPS is
  active.  Issue #1072 by Michael Kaiser-Nyman.
* Fixed handling of pinned gems in lockfiles with versions.  Issue #1078 by
  Ian Ker-Seymer.
* Fixed handling of git@example:gem.git URIs.  Issue #1054 by Mogutan Mogu.
* Fixed handling of platforms retrieved from the dependencies API.  Issue
  #1058 and patch suggestion by tux-mind.
* RubyGems now suggests a copy-pasteable `gem pristine` command when
  extensions are missing.  Pull request #1057 by Shannon Skipper.
* Improved errors for long file names when packaging.  Pull request #1016 by
  Piotrek Bator.
* `gem pristine` now skips gems cannot be found remotely.  Pull request #1064
  by Tuomas Kareinen.
* `gem pristine` now caches gems to the proper directory.  Pull request #1064
  by Tuomas Kareinen.
* `gem pristine` now skips bundled gems properly.  Pull request #1064 by
  Tuomas Kareinen.
* Improved interoperability of Vagrant with RubyGems.  Pull request #1057 by
  Vít Ondruch.
* Renamed CONTRIBUTING to CONTRIBUTING.rdoc to allow markup.  Pull request
  #1090 by Roberto Miranda.
* Switched from #partition to #reject as only one collection is used.  Pull
  request #1074 by Tuomas Kareinen.
* Fixed installation of gems on systems using memory-mapped files.  Pull
  request #1038 by Justin Li.
* Fixed bug in Gem::Text#min3 where `a == b < c`.  Pull request #1026 by
  fortissimo1997.
* Fixed uninitialized variable warning in BasicSpecification.  Pull request
  #1019 by Piotr Szotkowski.
* Removed unneeded exception handling for cyclic dependencies.  Pull request
  #1043 by Jens Wille.
* Fixed grouped expression warning.  Pull request #1081 by André Arko.
* Fixed handling of platforms when writing lockfiles.

=== 2.4.4 / 2014-11-12
891d9dc
@brixen brixen added a commit that referenced this issue
@brixen brixen Updated RubyGems to 2.4.6.
RubyGems 2.4.6 installed

=== 2.4.6 / 2014-02-05

Bug fixes:

* Fixed resolving gems with both upper and lower requirement boundaries.
  Issue #1141 by Jakub Jirutka.
* Moved extension directory after require_paths to fix missing constant bugs
  in some gems with C extensions.  Issue #784 by André Arko, pull request
  #1137 by Barry Allard.
* Use Gem::Dependency#requirement when adding a dependency to an existing
  dependency instance.  Pull request #1101 by Josh Cheek.
* Fixed warning of shadowed local variable in Gem::Specification.  Pull request
  #1109 by Rohit Arondekar
* Gem::Requirement should always sort requirements before coercion to Hash.
  Pull request #1139 by Eito Katagiri.
* The `gem open` command should change the current working directory before
  opening the editor.  Pull request #1142 by Alex Wood.
* Ensure quotes are stripped from the Windows launcher script used to install
  gems.  Pull request #1115 by Youngjun Song.
* Fixed errors when writing to NFS to to 0444 files.  Issue #1161 by Emmanuel
  Hadoux.
* Removed dead code in Gem::StreamUI.  Pull request #1117 by mediaslave24.
* Fixed typos.  Pull request #1096 by hakeda.
* Relaxed CMake dependency for RHEL 6 and CentOS 6.  Pull request #1124 by Vít
  Ondruch.
* Relaxed Psych dependency.  Pull request #1128 by Vít Ondruch.

=== 2.4.5 / 2014-12-03

Bug fixes:

* Improved speed of requiring gems.  (Around 25% for a 60 gem test).  Pull
  request #1060 by unak.
* RubyGems no longer attempts to look up gems remotely with the --local flag.
  Pull request #1084 by Jeremy Evans.
* Executable stubs use the correct gem version when RUBYGEMS_GEMDEPS is
  active.  Issue #1072 by Michael Kaiser-Nyman.
* Fixed handling of pinned gems in lockfiles with versions.  Issue #1078 by
  Ian Ker-Seymer.
* Fixed handling of git@example:gem.git URIs.  Issue #1054 by Mogutan Mogu.
* Fixed handling of platforms retrieved from the dependencies API.  Issue
  #1058 and patch suggestion by tux-mind.
* RubyGems now suggests a copy-pasteable `gem pristine` command when
  extensions are missing.  Pull request #1057 by Shannon Skipper.
* Improved errors for long file names when packaging.  Pull request #1016 by
  Piotrek Bator.
* `gem pristine` now skips gems cannot be found remotely.  Pull request #1064
  by Tuomas Kareinen.
* `gem pristine` now caches gems to the proper directory.  Pull request #1064
  by Tuomas Kareinen.
* `gem pristine` now skips bundled gems properly.  Pull request #1064 by
  Tuomas Kareinen.
* Improved interoperability of Vagrant with RubyGems.  Pull request #1057 by
  Vít Ondruch.
* Renamed CONTRIBUTING to CONTRIBUTING.rdoc to allow markup.  Pull request
  #1090 by Roberto Miranda.
* Switched from #partition to #reject as only one collection is used.  Pull
  request #1074 by Tuomas Kareinen.
* Fixed installation of gems on systems using memory-mapped files.  Pull
  request #1038 by Justin Li.
* Fixed bug in Gem::Text#min3 where `a == b < c`.  Pull request #1026 by
  fortissimo1997.
* Fixed uninitialized variable warning in BasicSpecification.  Pull request
  #1019 by Piotr Szotkowski.
* Removed unneeded exception handling for cyclic dependencies.  Pull request
  #1043 by Jens Wille.
* Fixed grouped expression warning.  Pull request #1081 by André Arko.
* Fixed handling of platforms when writing lockfiles.

------------------------------------------------------------------------------
c3b4f95
@brixen
Rubinius member
@brixen brixen closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.