Undefined symbol when using ctypes.foreign

[Hirrolot]

Run dune init proj --kind=library foo. Put the following support.c file in lib/:

#include <stdio.h>

void foo(void) {
    puts("foo");
}

lib/dune must look as follows:

(library
 (public_name foo)
 (name foo)
 (libraries ctypes.foreign)
 (foreign_stubs
  (language c)
  (names support)))

lib/Foo.ml must be:

open Ctypes
open Foreign

let test = foreign "foo" (void @-> returning void)

Type dune build && dune install.

Then create a new project with dune init proj --kind=executable test with the following bin/dune:

(executable
 (public_name test)
 (name main)
 (libraries test foo))

and the following bin/main.ml:

open Foo

let () = test ()

Type dune exec test and see the following error:

Fatal error: exception Dl.DL_error("_build/install/default/bin/test: undefined symbol: foo")

I expected the code to print foo and exit.

Specifications

$ dune --version
3.15.0
$ ocamlc --version
5.1.2+dev0-2023-12-7

My OS is Ubuntu 22.04 jammy (x86_64 Linux 6.5.0-28-generic).

[Hirrolot]

It is also interesting to note that calling Foo.test works in the REPL as expected. The problem seems to be with Dune specifically.

[emillon]

This comes from a confusion between dynamic loading and dynamic linking.
ctypes.foreign is a method to do dynamic loading (something like a plugin system). When you don't pass ~from to Foreign.foreign, it looks for the dynamic symbols in the current address space, which will include dynamically linked libraries.
In your case:

• the native dynamic linker is able to determine that your main executable doesn't use any dynamic symbol from support.so, so it's not bringing it into the address space (this behaviour is called --as-needed and has become the default in many distributions)
• the bytecode runtime system, including the one used by utop, isn't as sophisticated and will keep support.so as a dynamic dependency even though there's no dynamic symbol reference to it.

So the fact that it works in bytecode mode is more of an unintended side effect. NB: it's not really a Dune issue, if you compile manually with ocamlopt and friends you'll see the same behavior.

To fix this, you have several alternatives:

• you could switch to stub generation instead of using ctypes.foreign. this will create C and ocaml code that you'll link "normally" to your application, so there's no dynamic loading involved at runtime, just like a normal binding. Dune has some experimental support for this which you can try by following this document.
• force the link to be kept by declaring a dynamic symbol dependency in the library, like external foo : void -> void (the types don't have to match, and don't call this function). This might not be portable.
• if this is indeed some kind of plugin / optional behavior, you can make it use dynamic loading the "right" way: call Dl.dlopen with a path to the .so file, and pass a ~from argument to Foreign.foreign (you'll have to determine how to pass these paths at runtime)
• you could change the linker flags to pass --no-as-needed. I wouldn't recommend this because your project will be harder to package / will not work on some distributions.

Thanks

[Hirrolot]

Thanks for your response. However, if I define foreign_stubs in a Dune executable project that calls Foo.test, everything works. Why's that?

[emillon]

What does the dune file look like?

[Hirrolot]

Take the project foo for example. test/dune will look as follows:

(test
 (name test_foo)
 (libraries foo)
 (foreign_stubs
  (language c)
  (names support)))

With support.c located in test/.

lib/dune will look as follows:

(library
 (public_name foo)
 (name foo)
 (libraries ctypes.foreign))

and test/test_foo.ml:

let () = Foo.test()

After dune test, I see foo printed in the terminal, as expected.

[emillon]

In that case, support.o is statically linked into the test executable (you can see the .o file being passed in _build/log).

[Hirrolot]

So Dune's approach is to link libraries dynamically if stubs are defined in lib/dune, and link statically if stubs are defined in test/dune? What's the reasoning behind this behaviour? (And can I achieve static linking of my support.c when it's defined in the library's stubs?)

[emillon]

Dune isn't really taking a decision for library authors here: it's calling ocamlmklib to build static and shared versions of your stubs, so end consumers can either create static or dynamic executables.
I know that it might seem arbitrary in this toy example but in general consider that:

• the library and executable might not live in the same repository; your users don't see your source code / intermediate files besides the compiled ocaml code and stublibs
• your C stubs are likely going to use some native libraries. most users will use dynamic linking if they get these from their package manager

[Hirrolot]

After some fiddling, I came upon this solution (lib/dune):

(library
 (public_name foo)
 (name foo)
 (libraries ctypes.foreign)
 (extra_objects support))

(rule
 (targets support.o)
 (deps support.c)
 (action
  (run ocamlopt %{deps})))

The object file support.o is linked statically with the library, so there's no need to do anything else. Executables depending on foo work as expected.

[nojb]

So Dune's approach is to link libraries dynamically if stubs are defined in lib/dune, and link statically if stubs are defined in test/dune? What's the reasoning behind this behaviour?

Just to add a comment to @emillon's reply: when you define a library with C stubs, both static and dynamic versions of it are produced; one of the reasons for this is that while native executables are linked statically, bytecode executables need to load native libraries dynamically.

[Hirrolot]

Update: unfortunately, while my above solution works with tests, it doesn't work with other executable projects:

File "bin/dune", line 3, characters 7-11:
3 |  (name main)
           ^^^^
/usr/bin/ld: cannot find lib/support.o: No such file or directory
collect2: error: ld returned 1 exit status
File "caml_startup", line 1:
Error: Error during linking (exit code 1)