Skip to content

Commit

Permalink
I've filled out the C extension guide. It follows on from the 'make y…
Browse files Browse the repository at this point in the history 
…our own gem' guide.

The conventions are drawn from this tutorial
http://tenderlovemaking.com/2009/12/18/writing-ruby-c-extensions-part-1/
http://tenderlovemaking.com/2010/12/11/writing-ruby-c-extensions-part-2/
which I've used to create a couple of C extensions that seem to work well.
  • Loading branch information
@jdleesmiller @cldwalker
jdleesmiller authored and cldwalker committed Aug 26, 2011
1 parent 5477aa9 commit 23d65e2
Showing 1 changed file with 370 additions and 2 deletions.
372 changes: 370 additions & 2 deletions c-extensions.md
View file
Expand Up @@ -5,6 +5,374 @@ previous: /run-your-own-gem-server
next: /resources
---

How to make a gem with C extensions.
This guide explains how to package a C extension as a rubygem.

To get started, we'll add a C method to the `hola` gem from the
[make your own gem](/make-your-own-gem) guide. Then there's some general
[advice](#advice) and some [references](#references) for further reading.

<a id="tutorial"> </a>
Tutorial: Add a C extension to `hola`
========================================

At the end of the [make your own gem](/make-your-own-gem) guide, `hola` looked
like this:

% tree
.
|-- bin
| `-- hola
|-- hola.gemspec
|-- lib
| |-- hola
| | `-- translator.rb
| `-- hola.rb
|-- Rakefile
`-- test
`-- test_hola.rb

Now we'll add in a simple C extension.

<a id="tutorial-ext"> </a>
Create an `ext` folder
----------------------

Just as we put the ruby source files in the `lib/hola` folder, we put the C
extension source files in `ext/hola`. This folder contains two files:

% tree ext
ext
`-- hola
|-- extconf.rb
`-- hola.c

`extconf.rb` tells rubygems how to build the extension. For `hola`, it reads:

% cat ext/hola/extconf.rb
require 'mkmf'

create_makefile('hola/hola')

`mkmf` is part of ruby's standard library; it automates the process of creating
a [Makefile](http://en.wikipedia.org/wiki/Makefile) which, in turn, automates
the process of building the extension. So, when rubygems installs a gem, it
first runs the gem's `extconf.rb`, and then it runs `make` on the resulting
`Makefile`.

The output from `make` is a [dynamically linked
library](http://en.wikipedia.org/wiki/Shared_object). If we build `hola` on
Linux, the output is a file called `hola/hola.so`, where `.so` stands for
'shared object'. The extension is platform-dependent; it would be `.dll` on
Windows, or `.dylib` on Mac OS X. The name is what we pass to `create_makefile`.

In general, a good convention for the name is `<gem_name>/<gem_name>`. While it
is somewhat annoying to repeat the gem name in so many places, it will be seen
that it helps to avoid [naming conflicts](#naming), which are even more
annoying.

The C source code for the extension is in `hola.c`, which reads:

% cat ext/hola/hola.c
#include <ruby.h>

/* our new native method; it just returns
* the string "bonjour!" */
static VALUE hola_bonjour(VALUE self) {
return rb_str_new2("bonjour!");
}

/* ruby calls this to load the extension */
void Init_hola(void) {
/* assume we haven't yet defined Hola */
VALUE klass = rb_define_class("Hola",
rb_cObject);

/* the hola_bonjour function can be called
* from ruby as "Hola.bonjour" */
rb_define_singleton_method(klass,
"bonjour", hola_bonjour, 0);
}

`ruby.h` provides declarations for ruby's C API, which is where the `rb_...`
methods come from. The `Makefile` generated by `mkmf` ensures that the compiler
knows where this header file is. Some [references](#references) for ruby's C API
are given at the end of this guide.

Again, the name of the file is important, because it matches the second `hola`
in the `hola/hola` that we passed to `create_makefile` in `extconf.rb`.
Similarly, the name of the `Init_hola` function is important, because ruby looks
for a function with this name when it loads `hola/hola.so`. A [summary of these
naming rules](#naming) is provided at the end of the guide.

<a id="tutorial-gemspec"> </a>
Modify the `gemspec`
--------------------

For rubygems to know that a gem contains a C extension, we have to tell it about
`extconf.rb`, and we have to include the C source files in the `files` list.

% cat hola.gemspec
Gem::Specification.new do |s|
... (other stuff) ...

s.files = Dir.glob('lib/**/*.rb') +
Dir.glob('ext/**/*{.c,.h}')
s.extensions = ['ext/hola/extconf.rb']
s.executables = ['hola']

... (other stuff) ...
end

Here we're computing the `files` list dynamically; when we run `gem build`,
rubygems will include all matching files in the gem. Rubygems automatically
adds the `extensions` and `executables` to the gem, so you don't have to list
them under `files`.

<a id="tutorial-load"> </a>
Load the extension
------------------

The final step is to require the shared object so that ruby will load the
extension with the rest of the gem. To do this, we simply require `hola/hola` at
the top of `lib/hola.rb`:

% cat lib/hola.rb
require 'hola/hola'

... (rest of file unchanged) ...

This works because rubygems copies the shared object from `ext` to `lib` when
the gem is installed. It seems a bit like magic, but we can now build and
install the gem to see what's going on:

% gem build hola.gemspec
Successfully built RubyGem
Name: hola
Version: 0.0.1
File: hola-0.0.1.gem

The `gem build` command creates a `.gem` file from the `.gemspec`. It includes
all of the source files, but it doesn't compile the extension; that happens when
the gem is installed:

% gem install hola-0.0.1.gem
Building native extensions. This could take a while...
Successfully installed hola-0.0.1
1 gem installed

Where exactly the gem is installed to depends on how ruby is set up. On Linux
with [rvm](http://beginrescueend.com/), the ruby version manager, the gem is
installed into the following folder:

% tree ~/.rvm/gems/ruby-1.8.7-p352/gems/hola-0.0.1/
/home/john/.rvm/gems/ruby-1.8.7-p352/gems/hola-0.0.1/
|-- bin
| `-- hola
|-- ext
| `-- hola
| |-- extconf.rb
| |-- hola.c
| |-- hola.o
| |-- hola.so
| `-- Makefile
|-- lib
| |-- hola
| | |-- hola.so # <----
| | `-- translator.rb
| `-- hola.rb
`-- test
`-- test_hola.rb

We can see that the `ext/hola` folder contains our `extconf.rb` and `hola.c`
files, in addition to the `Makefile` generated during the install, the `hola.o`
object file generated by the compiler, and the finished product, `hola.so`. We
can also see that there is a copy of `hola.so` in `lib/hola`.

When we require the gem, rubygems puts `hola`'s `lib` folder on the
`$LOAD_PATH`, which is where `require` looks for files:

% irb -rubygems
ruby-1.8.7-p352 :001 > require 'hola'
=> true
ruby-1.8.7-p352 :002 > puts $LOAD_PATH
/home/john/.rvm/gems/ruby-1.8.7-p352/gems/hola-0.0.1/lib
... (lots of standard paths) ...
.
=> nil

When it sees `require 'hola/hola'` at the top of `lib/hola.rb`, it finds
`~/.rvm/gems/ruby-1.8.7-p352/gems/hola-0.0.1/lib/hola/hola.so`.
(Note that we don't have to write the `.so` extension -- ruby figures this out
for itself, which is fortunate, because it is platform-dependent.)

Finally, we can call our C extension's `bonjour` method:

ruby-1.8.7-p352 :003 > Hola.bonjour
=> "bonjour!"

The string `"bonjour!"` came from our C extension. Hooray!

Of course, we should also add a test to our test suite:

% cat test/test_hola.rb
require 'test/unit'
require 'hola'

class HolaTest < Test::Unit::TestCase

... (other tests) ...

def test_bonjour
assert_equal "bonjour!", Hola.bonjour
end
end

<a id="tutorial-rakefile"> </a>
Add helpful tasks to the `Rakefile`
-----------------------------------

Building and installing the gem every time you make a change quickly gets
tedious. To speed things up, it helps to add some extra tasks to your Rakefile
that automate the build process. The following should work on any unix, but
it would need some tweaking to run on Windows.

require 'rake/testtask'
require 'rake/clean'

NAME = 'hola'

# rule to build the extension: this says
# that the extension should be rebuilt
# after any change to the files in ext
file "lib/#{NAME}/#{NAME}.so" =>
Dir.glob("ext/#{NAME}/*{.rb,.c}") do
Dir.chdir("ext/#{NAME}") do
# this does essentially the same thing
# as what rubygems does
ruby "extconf.rb"
sh "make"
end
cp "ext/#{NAME}/#{NAME}.so", "lib/#{NAME}"
end

# make the :test task depend on the shared
# object, so it will be built automatically
# before running the tests
task :test => "lib/#{NAME}/#{NAME}.so"

# use 'rake clean' and 'rake clobber' to
# easily delete generated files
CLEAN.include('ext/**/*{.o,.log,.so}')
CLEAN.include('ext/**/Makefile')
CLOBBER.include('lib/**/*.so')

# the same as before
Rake::TestTask.new do |t|
t.libs << 'test'
end

desc "Run tests"
task :default => :test

Now typing `rake` will run the tests after building (or rebuilding) the
extension, as necessary:

% rake
(in /home/john/rubygems_hola)
/home/john/.rvm/rubies/ruby-1.8.7-p352/bin/ruby extconf.rb
creating Makefile
make
gcc -I. -I/home/john/.rvm/rubies/ruby-1.8.7-p352/lib/ruby/1.8/i686-linux -I/home/john/.rvm/rubies/ruby-1.8.7-p352/lib/ruby/1.8/i686-linux -I. -D_FILE_OFFSET_BITS=64 -fPIC -g -O2 -fPIC -c hola.c
gcc -shared -o hola.so hola.o -L. -L/home/john/.rvm/rubies/ruby-1.8.7-p352/lib -Wl,-R/home/john/.rvm/rubies/ruby-1.8.7-p352/lib -L. -rdynamic -Wl,-export-dynamic -Wl,-R -Wl,/home/john/.rvm/rubies/ruby-1.8.7-p352/lib -L/home/john/.rvm/rubies/ruby-1.8.7-p352/lib -lruby -lrt -ldl -lcrypt -lm -lc
cp ext/hola/hola.so lib/hola
Loaded suite /home/john/.rvm/gems/ruby-1.8.7-p352/gems/rake-0.8.7/lib/rake/rake_test_loader
Started
....
Finished in 0.00182 seconds.

4 tests, 4 assertions, 0 failures, 0 errors

If the C source and `extconf.rb` build script have not changed, then running
`rake` a second time runs only the test suite.

The code for this tutorial is [available on
github](https://github.com/jdleesmiller/hola).

Advice
======

<a id="naming"> </a>
Extension Naming
----------------

To avoid unintended interactions between gems, it's a good idea for each gem to
keep all of its files in a single folder. This is the motivation behind the
naming conventions used in the [tutorial](#tutorial). To summarize, the
suggested conventions for a gem with name `$g` are:

1. `ext/$g` is the folder that contains the source files and `extconf.rb`
1. `ext/$g/$g.c` is the main source file (there may be others)
1. `ext/$g/$g.c` contains a function `Init_$g`
1. `ext/$g/extconf.rb` calls `create_makefile('$g/$g')`
1. the gemspec sets `extensions = ['ext/$g/extconf.rb']` and sets `files` to
list any C source or header files in `ext/$g`
1. the first require in `lib/$g.rb` is `require '$g/$g'`

An alternative is to name the extension like `<gem_name>_ext` instead of
`<gem_name>/<gem_name>`. The result is that the `<gem_name>_ext.so` file is
installed into the gem's `lib` folder, and it can be required from
`lib/<gem_name>.rb` as `require '<gem_name>_ext'`. This also works, though it is
perhaps not as clean as the first convention.

Wrapping Existing Libraries
---------------------------

A common reason for writing a C extension is to wrap an existing C or C++
library. This can be done manually (see this tutorial --
[part
1](http://tenderlovemaking.com/2009/12/18/writing-ruby-c-extensions-part-1)
and [part
2](http://tenderlovemaking.com/2010/12/11/writing-ruby-c-extensions-part-2)),
but several tools also exist:

* [SWIG](http://www.swig.org/), the Simplified Wrapper Interface Generator, is
mature and probably the most popular
* [rb++](http://rbplusplus.rubyforge.org/) is nicer in several ways but is
less stable

Multi-Platform Extensions
-------------------------

The focus of this guide has been on writing extensions for Linux, but it is also
possible to write extensions that work on multiple operating systems.

This section needs help! [Please contribute.](http://github.com/rubygems/guides)

Multi-Implementation Extensions
-------------------------------

There are several ruby implementations. C extensions that use the ruby C API can
be loaded by the standard ruby interpreter (the MRI -- Matz's Ruby
Interpreter) and other C-based interpreters, but they cannot be loaded into
[JRuby](http://jruby.org/) (ruby on the Java Virtual machine) or
[IronRuby](http://ironruby.net/) (ruby on the Common Language Runtime (.NET)),
for example.

See [ruby-ffi](http://github.com/ffi/ffi) for a way to build extensions that
work with other Ruby implementations.

This section needs help! [Please contribute.](http://github.com/rubygems/guides)

References
==========

This guide is based largely on this excellent two-part tutorial:

* [part 1](http://tenderlovemaking.com/2009/12/18/writing-ruby-c-extensions-part-1)
* [part 2](http://tenderlovemaking.com/2010/12/11/writing-ruby-c-extensions-part-2)

The main references for ruby's C API are:

* [a chapter in the Pickaxe book](http://www.ruby-doc.org/docs/ProgrammingRuby/html/ext_ruby.html)
* [the README.EXT file](https://github.com/ruby/ruby/blob/trunk/README.EXT)

This guide needs help! [Contribute here.](http://github.com/rubygems/guides)

0 comments on commit 23d65e2

Please sign in to comment.