From 23d65e2acf534ed4d4b39935611991c47fa0eaa0 Mon Sep 17 00:00:00 2001 From: John Lees-Miller Date: Thu, 25 Aug 2011 14:04:24 +0100 Subject: [PATCH] I've filled out the C extension guide. It follows on from the 'make your own gem' guide. The conventions are drawn from this tutorial http://tenderlovemaking.com/2009/12/18/writing-ruby-c-extensions-part-1/ http://tenderlovemaking.com/2010/12/11/writing-ruby-c-extensions-part-2/ which I've used to create a couple of C extensions that seem to work well. --- c-extensions.md | 372 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 370 insertions(+), 2 deletions(-) diff --git a/c-extensions.md b/c-extensions.md index 64bdfe1c..2a0e9d1e 100644 --- a/c-extensions.md +++ b/c-extensions.md @@ -5,6 +5,374 @@ previous: /run-your-own-gem-server next: /resources --- -How to make a gem with C extensions. +This guide explains how to package a C extension as a rubygem. + +To get started, we'll add a C method to the `hola` gem from the +[make your own gem](/make-your-own-gem) guide. Then there's some general +[advice](#advice) and some [references](#references) for further reading. + + +Tutorial: Add a C extension to `hola` +======================================== + +At the end of the [make your own gem](/make-your-own-gem) guide, `hola` looked +like this: + + % tree + . + |-- bin + | `-- hola + |-- hola.gemspec + |-- lib + | |-- hola + | | `-- translator.rb + | `-- hola.rb + |-- Rakefile + `-- test + `-- test_hola.rb + +Now we'll add in a simple C extension. + + +Create an `ext` folder +---------------------- + +Just as we put the ruby source files in the `lib/hola` folder, we put the C +extension source files in `ext/hola`. This folder contains two files: + + % tree ext + ext + `-- hola + |-- extconf.rb + `-- hola.c + +`extconf.rb` tells rubygems how to build the extension. For `hola`, it reads: + + % cat ext/hola/extconf.rb + require 'mkmf' + + create_makefile('hola/hola') + +`mkmf` is part of ruby's standard library; it automates the process of creating +a [Makefile](http://en.wikipedia.org/wiki/Makefile) which, in turn, automates +the process of building the extension. So, when rubygems installs a gem, it +first runs the gem's `extconf.rb`, and then it runs `make` on the resulting +`Makefile`. + +The output from `make` is a [dynamically linked +library](http://en.wikipedia.org/wiki/Shared_object). If we build `hola` on +Linux, the output is a file called `hola/hola.so`, where `.so` stands for +'shared object'. The extension is platform-dependent; it would be `.dll` on +Windows, or `.dylib` on Mac OS X. The name is what we pass to `create_makefile`. + +In general, a good convention for the name is `/`. While it +is somewhat annoying to repeat the gem name in so many places, it will be seen +that it helps to avoid [naming conflicts](#naming), which are even more +annoying. + +The C source code for the extension is in `hola.c`, which reads: + + % cat ext/hola/hola.c + #include + + /* our new native method; it just returns + * the string "bonjour!" */ + static VALUE hola_bonjour(VALUE self) { + return rb_str_new2("bonjour!"); + } + + /* ruby calls this to load the extension */ + void Init_hola(void) { + /* assume we haven't yet defined Hola */ + VALUE klass = rb_define_class("Hola", + rb_cObject); + + /* the hola_bonjour function can be called + * from ruby as "Hola.bonjour" */ + rb_define_singleton_method(klass, + "bonjour", hola_bonjour, 0); + } + +`ruby.h` provides declarations for ruby's C API, which is where the `rb_...` +methods come from. The `Makefile` generated by `mkmf` ensures that the compiler +knows where this header file is. Some [references](#references) for ruby's C API +are given at the end of this guide. + +Again, the name of the file is important, because it matches the second `hola` +in the `hola/hola` that we passed to `create_makefile` in `extconf.rb`. +Similarly, the name of the `Init_hola` function is important, because ruby looks +for a function with this name when it loads `hola/hola.so`. A [summary of these +naming rules](#naming) is provided at the end of the guide. + + +Modify the `gemspec` +-------------------- + +For rubygems to know that a gem contains a C extension, we have to tell it about +`extconf.rb`, and we have to include the C source files in the `files` list. + + % cat hola.gemspec + Gem::Specification.new do |s| + ... (other stuff) ... + + s.files = Dir.glob('lib/**/*.rb') + + Dir.glob('ext/**/*{.c,.h}') + s.extensions = ['ext/hola/extconf.rb'] + s.executables = ['hola'] + + ... (other stuff) ... + end + +Here we're computing the `files` list dynamically; when we run `gem build`, +rubygems will include all matching files in the gem. Rubygems automatically +adds the `extensions` and `executables` to the gem, so you don't have to list +them under `files`. + + +Load the extension +------------------ + +The final step is to require the shared object so that ruby will load the +extension with the rest of the gem. To do this, we simply require `hola/hola` at +the top of `lib/hola.rb`: + + % cat lib/hola.rb + require 'hola/hola' + + ... (rest of file unchanged) ... + +This works because rubygems copies the shared object from `ext` to `lib` when +the gem is installed. It seems a bit like magic, but we can now build and +install the gem to see what's going on: + + % gem build hola.gemspec + Successfully built RubyGem + Name: hola + Version: 0.0.1 + File: hola-0.0.1.gem + +The `gem build` command creates a `.gem` file from the `.gemspec`. It includes +all of the source files, but it doesn't compile the extension; that happens when +the gem is installed: + + % gem install hola-0.0.1.gem + Building native extensions. This could take a while... + Successfully installed hola-0.0.1 + 1 gem installed + +Where exactly the gem is installed to depends on how ruby is set up. On Linux +with [rvm](http://beginrescueend.com/), the ruby version manager, the gem is +installed into the following folder: + + % tree ~/.rvm/gems/ruby-1.8.7-p352/gems/hola-0.0.1/ + /home/john/.rvm/gems/ruby-1.8.7-p352/gems/hola-0.0.1/ + |-- bin + | `-- hola + |-- ext + | `-- hola + | |-- extconf.rb + | |-- hola.c + | |-- hola.o + | |-- hola.so + | `-- Makefile + |-- lib + | |-- hola + | | |-- hola.so # <---- + | | `-- translator.rb + | `-- hola.rb + `-- test + `-- test_hola.rb + +We can see that the `ext/hola` folder contains our `extconf.rb` and `hola.c` +files, in addition to the `Makefile` generated during the install, the `hola.o` +object file generated by the compiler, and the finished product, `hola.so`. We +can also see that there is a copy of `hola.so` in `lib/hola`. + +When we require the gem, rubygems puts `hola`'s `lib` folder on the +`$LOAD_PATH`, which is where `require` looks for files: + + % irb -rubygems + ruby-1.8.7-p352 :001 > require 'hola' + => true + ruby-1.8.7-p352 :002 > puts $LOAD_PATH + /home/john/.rvm/gems/ruby-1.8.7-p352/gems/hola-0.0.1/lib + ... (lots of standard paths) ... + . + => nil + +When it sees `require 'hola/hola'` at the top of `lib/hola.rb`, it finds +`~/.rvm/gems/ruby-1.8.7-p352/gems/hola-0.0.1/lib/hola/hola.so`. +(Note that we don't have to write the `.so` extension -- ruby figures this out +for itself, which is fortunate, because it is platform-dependent.) + +Finally, we can call our C extension's `bonjour` method: + + ruby-1.8.7-p352 :003 > Hola.bonjour + => "bonjour!" + +The string `"bonjour!"` came from our C extension. Hooray! + +Of course, we should also add a test to our test suite: + + % cat test/test_hola.rb + require 'test/unit' + require 'hola' + + class HolaTest < Test::Unit::TestCase + + ... (other tests) ... + + def test_bonjour + assert_equal "bonjour!", Hola.bonjour + end + end + + +Add helpful tasks to the `Rakefile` +----------------------------------- + +Building and installing the gem every time you make a change quickly gets +tedious. To speed things up, it helps to add some extra tasks to your Rakefile +that automate the build process. The following should work on any unix, but +it would need some tweaking to run on Windows. + + require 'rake/testtask' + require 'rake/clean' + + NAME = 'hola' + + # rule to build the extension: this says + # that the extension should be rebuilt + # after any change to the files in ext + file "lib/#{NAME}/#{NAME}.so" => + Dir.glob("ext/#{NAME}/*{.rb,.c}") do + Dir.chdir("ext/#{NAME}") do + # this does essentially the same thing + # as what rubygems does + ruby "extconf.rb" + sh "make" + end + cp "ext/#{NAME}/#{NAME}.so", "lib/#{NAME}" + end + + # make the :test task depend on the shared + # object, so it will be built automatically + # before running the tests + task :test => "lib/#{NAME}/#{NAME}.so" + + # use 'rake clean' and 'rake clobber' to + # easily delete generated files + CLEAN.include('ext/**/*{.o,.log,.so}') + CLEAN.include('ext/**/Makefile') + CLOBBER.include('lib/**/*.so') + + # the same as before + Rake::TestTask.new do |t| + t.libs << 'test' + end + + desc "Run tests" + task :default => :test + +Now typing `rake` will run the tests after building (or rebuilding) the +extension, as necessary: + + % rake + (in /home/john/rubygems_hola) + /home/john/.rvm/rubies/ruby-1.8.7-p352/bin/ruby extconf.rb + creating Makefile + make + gcc -I. -I/home/john/.rvm/rubies/ruby-1.8.7-p352/lib/ruby/1.8/i686-linux -I/home/john/.rvm/rubies/ruby-1.8.7-p352/lib/ruby/1.8/i686-linux -I. -D_FILE_OFFSET_BITS=64 -fPIC -g -O2 -fPIC -c hola.c + gcc -shared -o hola.so hola.o -L. -L/home/john/.rvm/rubies/ruby-1.8.7-p352/lib -Wl,-R/home/john/.rvm/rubies/ruby-1.8.7-p352/lib -L. -rdynamic -Wl,-export-dynamic -Wl,-R -Wl,/home/john/.rvm/rubies/ruby-1.8.7-p352/lib -L/home/john/.rvm/rubies/ruby-1.8.7-p352/lib -lruby -lrt -ldl -lcrypt -lm -lc + cp ext/hola/hola.so lib/hola + Loaded suite /home/john/.rvm/gems/ruby-1.8.7-p352/gems/rake-0.8.7/lib/rake/rake_test_loader + Started + .... + Finished in 0.00182 seconds. + + 4 tests, 4 assertions, 0 failures, 0 errors + +If the C source and `extconf.rb` build script have not changed, then running +`rake` a second time runs only the test suite. + +The code for this tutorial is [available on +github](https://github.com/jdleesmiller/hola). + +Advice +====== + + +Extension Naming +---------------- + +To avoid unintended interactions between gems, it's a good idea for each gem to +keep all of its files in a single folder. This is the motivation behind the +naming conventions used in the [tutorial](#tutorial). To summarize, the +suggested conventions for a gem with name `$g` are: + +1. `ext/$g` is the folder that contains the source files and `extconf.rb` +1. `ext/$g/$g.c` is the main source file (there may be others) +1. `ext/$g/$g.c` contains a function `Init_$g` +1. `ext/$g/extconf.rb` calls `create_makefile('$g/$g')` +1. the gemspec sets `extensions = ['ext/$g/extconf.rb']` and sets `files` to + list any C source or header files in `ext/$g` +1. the first require in `lib/$g.rb` is `require '$g/$g'` + +An alternative is to name the extension like `_ext` instead of +`/`. The result is that the `_ext.so` file is +installed into the gem's `lib` folder, and it can be required from +`lib/.rb` as `require '_ext'`. This also works, though it is +perhaps not as clean as the first convention. + +Wrapping Existing Libraries +--------------------------- + +A common reason for writing a C extension is to wrap an existing C or C++ +library. This can be done manually (see this tutorial -- +[part +1](http://tenderlovemaking.com/2009/12/18/writing-ruby-c-extensions-part-1) +and [part +2](http://tenderlovemaking.com/2010/12/11/writing-ruby-c-extensions-part-2)), +but several tools also exist: + +* [SWIG](http://www.swig.org/), the Simplified Wrapper Interface Generator, is + mature and probably the most popular +* [rb++](http://rbplusplus.rubyforge.org/) is nicer in several ways but is + less stable + +Multi-Platform Extensions +------------------------- + +The focus of this guide has been on writing extensions for Linux, but it is also +possible to write extensions that work on multiple operating systems. + +This section needs help! [Please contribute.](http://github.com/rubygems/guides) + +Multi-Implementation Extensions +------------------------------- + +There are several ruby implementations. C extensions that use the ruby C API can +be loaded by the standard ruby interpreter (the MRI -- Matz's Ruby +Interpreter) and other C-based interpreters, but they cannot be loaded into +[JRuby](http://jruby.org/) (ruby on the Java Virtual machine) or +[IronRuby](http://ironruby.net/) (ruby on the Common Language Runtime (.NET)), +for example. + +See [ruby-ffi](http://github.com/ffi/ffi) for a way to build extensions that +work with other Ruby implementations. + +This section needs help! [Please contribute.](http://github.com/rubygems/guides) + +References +========== + +This guide is based largely on this excellent two-part tutorial: + +* [part 1](http://tenderlovemaking.com/2009/12/18/writing-ruby-c-extensions-part-1) +* [part 2](http://tenderlovemaking.com/2010/12/11/writing-ruby-c-extensions-part-2) + +The main references for ruby's C API are: + +* [a chapter in the Pickaxe book](http://www.ruby-doc.org/docs/ProgrammingRuby/html/ext_ruby.html) +* [the README.EXT file](https://github.com/ruby/ruby/blob/trunk/README.EXT) -This guide needs help! [Contribute here.](http://github.com/rubygems/guides)