New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
compile time gem dependencies #82
Comments
Current blocker (both on my Ubuntu host machine, and on rake-compiler-dev-box):
Relevant portion of the mkmf.log:
Reproduction instructions for Then, from a fresh 'vagrant up':
|
Hello Sam, First apologies for the lack of response, but I normally work on Open Source To better help you out, I'm going to over a few things I noticed from your First thing first, dependencies.As you mention before, it is clear your project depends on Nokogiri, but not Making that dependency to Nokogiri internals also pushes on you a dependency https://github.com/rubys/nokogumbo/blob/master/extconf.rb#L5 Next, you lookup for Nokogiri installation to determine it's source code Please keep this in mind as I move forward the other points. The last part of the dependencies is the availability of gumbo parser source Gem structureI couldn't miss noticing that your project is all the files in the root folder, I did notice that you copy files around in your Rakefile to be able to generate the gem structure before packaging. I wonder if wouldn't be better have that gem structure from the beginning and One thing that I find important is consistency and conventions. One thing you If you look at rake-compiler structure recommendation will see what I mean. This will greatly help both your project organization and welcoming other Build chain and compiling dependenciesAs mentioned before, having Nokogiri installation as gem to be able to This means that compiling natively will require nokogiri be installed but Cross-compiled Ruby cannot be used to install or compile gems, simply because Because of this, you cannot install a gem inside the cross-compiled Ruby that This could be avoided if instead of looking up for Nokogiri source code using The second limitation is libxml2 and libxslt, which are two dependencies of In your case, you're not dealing at all with such dependency, which will cause If you look at Nokogiri, they deal with libxml dependency in their extconf: https://github.com/sparklemotion/nokogiri/blob/master/ext/nokogiri/extconf.rb Last on this topic, gumbo-parser should be compiled as another dependency, Copying gumbo source files, as covered in your Either it is a submodule that is checked out and the source code part of the Cross compiling to WindowsOne of rake-compiler abilities is allow developers to cross-compile gems to While that might sound like a miracle, it just a sequence of conventions and To make things a bit more easier, a VM has been provided, which covers most In relation to the platforms questions, RubyInstaller (the official installers) The usage of In your particular case, these automated scripts might not serve you as 1.8.x As said before, the VM follows the standards and the existing conventions, but But for all these tools to works properly, all the previous points needs to Present to developers and the tools a standard/conventional gem structure, so Deal with the external dependencies either as modules of your project, I would suggest focusing on be able to generate a native gem (not even Of course, all this is my personal opinion on how to deal with these Hope it helps. |
Not a problem. Thanks for taking the time to respond.
Actually, my code depends heavily on libxml2 and gumbo (calling a number of public interfaces on each), and when done makes a single call to nokogiri.
Gumbo is new and probably won't be installed on most people's machines. However, it is straight C code, and extconf builds a Makefile that will compile it just fine, so my Rakefile will do a git clone and copy the necessary files into the ext/nokogumboc directory if it isn't already present. Note that this will result in a single .so (or .dll) file which embeds the parser.
Since I'm (optionally) copying files into the gem structure, that makes rake targets like clean and clobber more complicated if those directories contain both nokogumbo source files and other files.
This does not makes sense to me as installing a gem is merely a matter of putting the right files into the right places.
I seem to be finding the libxml2 headers (I develop on Ubuntu, and most of the headaches that nokogiri's build process appears to be working around is due to Mac OSX issues), that being said, you probably are right when it comes to link time.
This seems to be a sticking point for you, so lets start with that. Lets put aside libxml2 and nokogiri for a moment, and consider gumbo_parser. Everything I need is in the src directory. All I need to add is an extconf.rb, and in that file specify Given that as the requirements, can I ask how you would recommend I structure my repository so that the gumbo parser will not only be compiled but also installed? |
But remains true what I said: you depend on nokogiri internals. Also, you're ignoring the point I mention about libxml. While it might be
Again goes to the same point I mentioned: compilation of the library. By I gave you some pointers on some projects that compile dependencies, rugged
I tried to get your work running on my machine prior making this comment. On other projects that I got request to be involved, I freely and happily However in this case, the way things are build aren't solid enough for me
But that covers just the gem, not the dependencies. Can you point me where Again, please see the work done in nokogiri to cross compile libxml, which The cross compiled ruby cannot be executed natively, that is why the gem You installing Ubuntu libxml dependencies only solves the native For cross compilation to work, ruby and the dependencies of your library English is not my native language, so perhaps what I'm explaining do not
Please read my comments above, including the source might not serve to All my previous comments were made on the base that I was not able to get a
I believe in my previous comment included links to rubygems guides, Something that I learned over the years is that instead of doing the Cross compilation is neither simple nor have a unique solution, I'm sharing Regards. Sorry for top posting. Sent from mobile.
|
Gumbo is a pure C99 library with no outside dependencies.
Those appear to presume that you have installed the necessary dependencies separately, and point to where they are installed. Perhaps I am attempting to be too clever, but as gumbo is a pure C library, I'm instead cloning the repository and copying what I need into the ext/nokogumboc directory prior to running extconf.rb.
Again, I may trying to be too clever, but I let nokogiri do that for me:
You are doing well! And my high school Spanish, while rusty, was good enough to follow this presentation: http://blog.mmediasys.com/2011/11/26/rubyconf-argentina-and-fenix/ :-)
I'm still not getting it :-( That's why I hoped that we could start with something simpler at first. Perhaps with an an earlier revision of nokogumbo? Note that this /nearly/ follows the recommended structure (I didn't know then to insert 'nokogumboc' into the ext directory structure). It consists of a single C file that only has compile time dependencies against Ruby and Gumbo -- and again the latter is a pure C99 library with no other dependencies.
I can appreciate that. But hopefully we can find a happy medium between you doing all of the work and me trying clumsily to follow the examples I find and for you to tell me to once again look at those same examples. Taken as a whole, gumbo + nokogumbo is 12 C99 source files and 14 header files that depend only on Ruby. What I will try to do (probably in a separate branch) is to get just that working before I attempt to tackle pulling in nokogiri and libxml2. |
You're correct, I was actually talking any possible special library that Gumbo might require when setting up on Windows, but seems Gumbo doesn't use anything in particular (by looking at autoconf and friends), so just including the code will be ok.
Perhaps gumbo repository can be used as submodule and then added to the list of objects instead of copied over? I normally try to avoid do that and instead produce a static library of the dependency (gumbo in this case) and link against it. That is what Rugged does with libgit2. I would say that taking a step back and going with a simpler (non-nokogiri) approach first might be better than using Nokogiri internals. I need to push some commits for tiny_tds project first and then will take a look to your early structure approach. Will send my comments later today. Thank you. |
I've added it as a submodule. I can't seem to find documentation for mkmf that covers all of the things you can do with global variables, but from what I can tell, mkmf assumes that everything is in one directory. (You can specify which directory you want to use, but you can't specify multiple). I could be wrong about this, as this is based on reviewing the source code.
libgit2 provides Makefile.embed. gumbo-parser builds this using ./configure. Not being sure whether or not ./configure would play nice with cross compiling, at the moment I'm sticking with extconf and mkmf.
Such an approach would come with a significant CPU and memory usage penalty, but I've added conditional compilation instructions falling back to such an approach should nokogiri headers not be found. Current statusAt this point in time, nokogumbo is effectively a "pure C" library with no required dependencies other than Ruby itself. As such, I would think that cross compiling should be easy peasy. Unfortunately, I'm still seeing this error. This is both on my machine and on rake-compiler-dev-box, The error indicates that winsock can't be linked in to the library -- something I don't explicitly require. Building and testing on Ubuntu or Mac OSX is as simple as installing dependencies (via Running |
Success?Apparently, I was misunderstanding the error message produced... the problem was libxml2 not being found. But as libxml2 is not a hard requirement any more, I rearranged my extconf.rb file and am able to build Next up: figuring out how to build a Gem for Windows. |
Not quite successful yet:
Also, it doesn't appear that mingw supports C99. :-( So, current status is that now I have implemented the recommended directory structure, reference gumbo-parser as a submodule, have no required dependencies beyond Ruby, and while this appears to be sufficient to install the gem on Windows 8 with RailsInstaller, I still can't cross compile. |
Hello @rubys, sorry for the late response, but I was down the rabbithole for the past months at work. Shared objects Perhaps you tried to run nokogumbo after the cross compilation? If that was the case, you need to perform a native compile so the |
Any chance you can try? As I stated above, the current state is that nokogumbo can now be compiled with no required dependencies beyond Ruby, and I have implemented the recommended directory structure, yet I can not get it to work. As such, it should be the perfect candidate project for rake-compiler; but I've tried everything and failed. |
@rubys will try again this weekend while I work fixing some issues with rake-compiler and rake-compiler-dev-box. |
Background:
Nokogumbo provides the ability for a Ruby program to invoke the Gumbo HTML5 parser and to access the result as a Nokogiri::HTML::Document.
Nokogumbo makes use of a single Nokogiri API:
Nokogumbo successfully builds and runs today on Ubuntu Linux and OSX Mountain Lion. I recently converted the Rakefile to use rake-compiler in anticipation of cross compiling to Windows. My next step is to update my extconf.rb as follows:
The result, predictably, is:
I'm also looking into rake-compiler-dev-box, and hitting all sorts of mundane issues (current example:
package_win32_fat_binary.sh
includes Ruby 1.8.7, but nokogiri no longer supports that release; runningrake cross compile
myself results inGem rake is not installed
; rake-compiler-dev-box clearly makes use of rvm, but doesn't set up the rvm command).I'm working through all of these (eg:
. .rvm/scripts/rvm
), and will contribute back whatever I learn in the form of pull requests for code, documentation, whatever. Meanwhile, any advice I can get would be appreciated.The text was updated successfully, but these errors were encountered: