Dynamic Libraries in C and C++

I first encountered software "plugins" with Photoshop 3.0 and Kai's Power Tools on Windows 3.1... and it blew my mind! The idea that a piece of software could be architected so that you could add and remove extra 3rd party modules and have them integrate so tightly was something of a revelation to me. Separate programs communicating was one thing, but this was a seamless extension to an existing piece of software. Amazing.

In programming this technique is known as a "dynamic library" and its implementation varies by operating system, largely splitting between Windows and Unix-variant lines. This is because it involves a great deal of behind-the-scenes magic in order to open the library, sift through its symbol table to find the function(s) you are looking for, and then finally invoke the new function(s) without disrupting the operation of your program. This process is known as 'linking' and it happens for most programs at run time as well, so it is natural that this is dependent on the operating system's understanding of libraries and how a running process is constructed.

I'm going to show how this process works in C and C++, providing simple examples along the way. Putting together sophisticated user interfaces that exploit these techniques is beyond the scope here, I want to just show the basics and leave the menus, etc. up to you. My examples are for *nix systems and I'm going to assume proficiency in C and C++.

---

Dynamic Libraries in C

The conceptual problem is this: from inside of a running program, how do I find a function in a compiled file and then how do I execute it?

The second part of that question is easily answered by the language itself. C and C++ provide a data type of "function pointer" that can be used syntactically in the same places that a function can, but can have an assignable value. Declaring a function pointer looks like void (*my_func_ptr)(); and you can call it just like any other function with my_func_ptr();. This declares a function, my_func_ptr, that accepts no arguments and returns nothing. If necessary, you can declare an argument list and return type, such as int (*my_func_ptr)(char *, const int); . In this case, my_func_ptr() takes a pointer to char and an int and returns an int (for example, for counting the number of vowels in the first n characters in a string and returning that value).

It's important to point out that C/C++ can only resolve this data typing at compile time. Since type information is not baked into compiled objects, there is no way for the program to determine at run time if the function has the correct signature. With dynamic libraries this is entirely up to you to make sure that you're invoking the dynamically loaded functions with the correct arguments and working with their output properly. This is why it's typical to put function signatures in .h files so that there is a published standard for interconnecting modules to conform to and the compiler can help sanity check the comings and goings of your function calls.

So if we can get the address of a function in a compiled file, we can use assign it to a function pointer. How do we get the address of a function in a compiled file, though? In Unix systems this is provided by the functions dlopen() and dlsym() (there are also dlclose() and dlerror(), which I'll leave for you to read the man pages of). dlopen() takes two arguments, a file path and a set of flags that indicate how the system should manage the contents of the library, and it returns a handle to the library. dlsym() takes the library handle that dlopen() provided and a string for a symbol (such as the name of a function), and returns... a function pointer! (or whatever datatype that symbol references) Now we have the mechanisms to see how this works in code, disregarding error checking:

#include <dlfcn.h>

void * library_handle;
void (* my_dyn_func)();
library_handle = dlopen("mylibrary.so", 0);  // 0 == default flags == RTLD_GLOBAL | RTLD_LAZY
my_dyn_func = dlsym(library_handle, "plugin_func_name");
my_dyn_func();

This isn't the whole story, though. In order for dlopen() and dlsym() to be able to read and understand the library ("mylibrary.so" in the example), you'll need to compile it with special flags to indicate that you want the compiler to preserve the symbol table and format it so that the loader can read it properly. With gcc this flag is "-shared", which stands for "shared object". This is where the .so filename extension comes from as well. Note that this is not an "object" in the sense of object-oriented programming.
It's just a generic use of the term to indicate a binary file that contains a set of related functions and data.

So that's it. Here's the bare minimum:

• compile your function into a dynamic library (aka shared object) with "-shared"
• open the library with dlopen() and find the function with dlsym()
• assign the value to a function pointer

There is a very simple example in the dlopen_c_example directory. It includes two library files (testlib1.c, testlib2.c), an application that can use either library based on its command line arguments (dlopen.c), and a Makefile. You can compile with:

# make

And try it out with:

# ./dlopen testlib1.so test

You can use "testlib1.so" or "testlib2.so" for the first argument or "testa" or "testb" for the second. This illustrates not just dynamic loading of different code at run time, but also running of different functions. I've also included finding a string that is the name of the library for the user's benefit. Try out all the combinations at once with:

# make test

---

Dynamic Libraries in C++

For C we simply wanted to execute a dynamically loaded function. Naturally in C++ we want to work with objects, ideally instantiating an object from a dynamically loaded class.

Now that we've covered how the mechanisms work in C, certainly C++ has a more sophisticated OO paradigm for dynamically loading code, right? Unfortunately, that's not the case. We still use dlopen() and dlsym(), but because of the way C++ mangles method names it is difficult or impossible to acquire the address of an object's constructor on the fly. This makes creating objects from dynamic libraries much more difficult.

One technique for managing this is to have a statically defined object in the dynamic library that acts as a factory for creating the object we have in mind. If we create a standard interface for this factory and a generic superclass for the object we want to create, we can subclass both while while still allowing the calling program to find the object and create the dynamic object. It sounds complicated... and it is! If we examine the technique one step at a time while looking at the necessity of each mechanism it should make the process clear.

We start by understanding the need and design of our factory class and why an object of this factory class needs to be instantiated statically in the compiled library, instead of dynamically at runtime in the code as you may be more familiar with.

In order to handle multiple methods with the same name but differing parameter types, as well as namespaces, C++ compilers will perform "name mangling" on the symbols that are stored in compiled object code. This name mangling is not standardized, and it can even vary between different versions of the same compiler! (There is an excellent example on Wikipedia in the article on name mangling). Because of this name mangling it is impossible to reach inside a shared object and pull out the address of an class constructor, even if you know the name of the class. For example, the constructor for a "Foo" object has the name "__ZN3fooC1Ev" in the symbol table on my laptop (GCC 4.2.1 on MacOS 10.6).

So if all method names including constructors are unknown, how can we make use of a C++ dynamic library? One solution, and the one I'll demonstrate, is to have a statically allocated factory object in the library that inherits from a common factory class. This common factory class contains a virtual method that we override with our own method that we'll call at run-time to create the dynamic object we ultimately want. What this means in code is that we declare an instance object of this factory class in file scope (outside of any methods/functions) so we are declaring it as static. This has two results: the dynamic library will be arranged so when dlopen() opens the library the constructor of this factory object is called, and most importantly that the name of the instantiated factory object is not mangled. This gives us a known anchor into the dynamic library for code that we can use to do more... such as creating other objects!

The use of a factory may seem a roundabout technique for being able to dynamically load in a new object, but as we walk through the example you may be inspired and see ways in which this technique provides a great deal of flexibility for managing dynamic objects of unknown class. For the example, the objects that we want to create dynamically at runtime are descended from a base class, plugin. The interface to plugin is defined as:

class plugin
{
    public:
        // default constructor
        virtual void test() { };
        virtual ~plugin() {};
};

Pretty straightforward. We rely on the compiler to create the default constructor, and test() will be the function that we override in our plugins that performs expanded functionality. The destructor is marked virtual because we expect full-featured plugins will have their own unloading requirements (removing menu entries, closing files or sockets, etc.) However, for the example the plugins will have very simple destructors.

Now we can look at the factory class. Remember that the purpose of the factory is to have a known function entry point in the library that we can find and use to create our objects. The factory can be made as complex as needed, but at its simplest it only provides a virtual function that returns an instance of "plugin", so its interface consists of a single virtual method:

class factory
{
    public:
        // default constructor
        virtual plugin * makedyn() { };
        // default destructor
};

We can now examine the code for a dynamic library that uses and extends these base classes. In it are defined two classes: the plugin class that our program will use and a factory that knows how to create this plugin. Also included in the file is a string that provides a name for the plugin and the statically declared instance of the plugin factory that we'll use to create a new instance of the plugin. Here's the code for the foo plugin:

const char * classname = "foo Object";  // name for this plugin, useful for user feedback

// the actual plugin class
class foo : public plugin
{
    public:
        foo()  {  cout << "    foo created [foo.cc - foo::foo()]" << endl;  };
        ~foo() {  cout << "    foo destroyed [foo.cc - foo::~foo()]" << endl;  };
        virtual void test() {  cout << "    foo tested [foo.cc - foo::test()]" << endl;  };
};


// the class of the plugin factory
class foofactory : factory
{
    public:
        foo * makedyn() {  cout << "    making foo [foo.cc foofactory::makedyn()]" << endl;  return new foo;  }
};


// a statically declared instance of our derived factory class
foofactory Factory;

Observe how the foofactory class overrides the makedyn() function to create and return a pointer to an object of class foo. Also note the foofactory instance named Factory, declared at the bottom of the code snippet. Finally, see that the foo class overrides the constructor, destructor and test() methods of the plugin parent class.

Finally, let's examine how a program can make use of this plugin. In the C example we declared a function pointer and then assigned to it the value of a function address once we had looked it up in the dynamic library, like this:

void (* my_dyn_func)();
library_handle = dlopen("mylibrary.so", 0);
my_dyn_func = dlsym(library_handle, "plugin_func_name");
my_dyn_func();

Since this is C++ and we're working with objects, instead of a function pointer we'll use pointers to objects to hold both the dynamic library's "entry point" as well as the plugin that contains the dynamic code we're loading. Now we can reveal the actual dlopen(), dlsym(), and factory method that loads our dynamic plugin class:

#include <dlfcn.h>

void * library_handle;
factory * myFactory;
plugin * myPlugin;

library_handle = dlopen("mylibrary.so", 0);

myFactory = (factory *)dlsym(handle, "Factory");  // find the statically declared factory
myPlugin = myFactory->makedyn();                  // create our dynamic plugin
myPlugin->test();                                 // call our plugin-specific code

The critical point in this code is the string "Factory" that dlsym() uses to find the statically defined factory class. Like "plugin_func_name" in the C example, there must be a variable with a known symbol name that dlsym() can find. Once the address of this symbol is found and cast to a factory class we can use its overriden virtual method makedyn() to create a plugin object.
Then we can call the overriden virtual method test() of the plugin object to call plugin-specific code.

And, like the C example, the plugin must be compiled with the "-shared" flag so that the object file includes the symbol table that allows it to be parsed and then found by the dlopen() and dlsym() functions. On most operating systems (but not MacOSX) an additional flag may need to be passed as well, "-fPIC". PIC stands for "position independent code" and it means that the resulting object code is structured so that function calls aren't hardwired to specific locations in memory, a requirement for code that may be loaded anywhere and in any order at run-time. A thorough explanation of PIC is outside the scope of this article but generally this means that the resulting assembly code uses relative addressing in function calls, rather than absolute addresses. Wikipedia has a good article on PIC.

That wraps up the C++ dynamic code example. Here's the bare minimum for this technique:

• two base classes: the plugin itself with virtual methods to be overridden in derived classes for custom functionality, and a factory with a single virtual method to be overridden by a plugin's corresponding derived factory class
• plugin code that extends these two base classes, with a statically declared instance of the derived factory in the plugin library
• compile your plugin code into a dynamic library (aka shared object) with "-shared" and "-fPIC"
• open the library with dlopen and find the factory instance with dlsym
• use the factory's virtual method to create a new plugin object

There is an example in the dlopen_cpp_example directory. It includes three plugins (foo.cc, bar.cc, and baz.cc), an application that can dynamically load a library based on command line arguments (dlopen.cc), headers, and a Makefile.
You can compile with:

# make

And try it out with

# ./dlopen foo.so

You can substitute foo.so with bar.so or baz.so and the code will load up the bar and baz plugins appropriately. Try all the combinations at once with:

# make test

Have fun! And if you have any questions, please feel free to comment below or drop me a line.