Rules for ctypes

[ghost]

Ctypes is the recommended way to write C stubs for OCaml, Jbuilder have builtin support for it. (related to #131).

[g2p]

I'd be interested in support for ctypes.foreign.
Listing non-stub C sources in c_names doesn't appear to do the trick.
Is there some linker magic I'm missing?

(Edit: yes I was. I needed -rdynamic in c_library_flags, and an external declaration to make sure the C function doesn't get eliminated as dead code)

[XVilka]

I tried to use the rules like:

(jbuild_version 1)
(library
  ((name mylib)
  (public_name mylib)
  (libaries (core ctypes ctypes.foreign))
  (c_library_flags (:standard -lz -lm -llzma -pthread))
  (library_flags (-cclib -Wl,-E))
  (self_build_stubs_archive (libmylib))
))
(rule
((targets (libmylib_stubs.a dllmylib_stubs.so))
  (deps ((files_recursively_in mylib)))
  (action (bash "
  ${MAKE} -s -C mylib clean
  $(MAKE} -s -C mylib libmylib.a
  $(MAKE} -s -C mylib libmylib.so
  cp mylib/libmylib.a libmylib_stubs.a
  cp mylib/libmylib.so dllmylib_stubs.so
  ${MAKE} -s -C mylib clean
"))))

But then when I use this library in any kind of application, it links but doesn't link the symbols from this "libmylib.{a,so}", raising the error when running:

Uncaught exception:
   Dl.DL_error("_build/default/myapp.exe: undefined symbol: "myfunction1")

To be honest, I have no idea why this is happens and why jbuilder can't handle this case.

[ghost]

Try (self_... (mylib)). This is what https://github.com/janestreet/re2/blob/master/src/re2_c/jbuild does and it works

[aantron]

I have the same problem as @XVilka.

The re2 workaround didn't work for me, and I didn't expect it to. re2 has a separate build script for library archive files. In my case, Jbuilder was already correctly building the C library. Adding (self...) just made Jbuilder report that it failed to find rules for building a file that it otherwise successfully built (libfoo_stubs.a). I also confirmed with nm and dlopen/dlsym that the stubs archive being built by Jbuilder does indeed export the symbol that could not be found.

Everything seems to work if I add a dummy external that depends from OCaml on a symbol from the stubs library, however.

It looks like Jbuilder+Ctypes is broken.

Also, to try to find more examples, I tried listing opam packages that depend on both ctypes-foreign and jbuilder, but found none.

[XVilka]

Just for the record, following jbuild file worked for me:

(jbuild_version 1)
(library
  ((name newlib)
   (public_name newlib)
   (wrapped false)
   (libraries (core ctypes ctypes.foreign))
   (c_library_flags (:standard -Wl,--no-as-needed -lz -lm -llzma -pthread))
   (library_flags ((-cclib -Wl,-E) (-cclib -Wl,--no-as-needed) (lib/libmylib_stubs.a)))
   (self_build_stubs_archive (libmylib))
   ))
(rule
((targets (libmylib_stubs.a dllmylib_stubs.so))
	(deps ((files_recursively_in mylib)))
	(action (bash "
	${MAKE} -s -C mylib clean
	${MAKE} -s -C mylib libmylib.a
	${MAKE} -s -C mylib libmylib.so
	cp mylib/liblibmylib.a libmylib_stubs.a
	cp mylib/libmylib.so dllmylib_stubs.so
	${MAKE} -s -C mylib clean
	"))))

[ghost]

It's possible that there is an issue with ctypes-foreign. It works fine with static stub generation though, async_ssl does this for instance.

[rgrinberg]

@diml given your recent change of approach to supporting external tools (implement a satisfying approach inside dune, generalize to plugins later). Do you think we can have good support for ctypes out of the box as well?

[ghost]

Absolutely

[paracetamolo]

Any news on the support in dune for ctypes?

[ghost]

Nothing has been done so far, but we are planning to look at this with @yallop

[vinczedani]

Any news on the support in dune for ctypes?

[yallop]

I think progress on ctypes support may be blocked by my lack of time at the moment. I can't really look at this directly at the moment, but I'm happy to answer questions, or provide feedback, if someone else wants to make a start.

Perhaps the dune/ctypes rules used in @avsm's yaml bindings would be a useful starting ponit.

[rgrinberg]

I'll also recommend the async_ssl bindings.

@vinczedani do you have any particular pain points when using ctypes/dune? We rely on user input to make improvements. AFAIK, ctypes + jbuilder work fine if you're using the stub generation, it's just a little boilerplate heavy.

@aantron would you happen to have a reproducible test case for us?

[aantron]

@rgrinberg Not at this point, it's out of my cache and I won't have time to recreate it soon.

[XVilka]

Just met the problem with manual workarounds - there is no easy way to compile ctypes-generated bindings code statically - C stubs still dynamically load libc, and binary needs libffi.

See also:

• Static linking ? yallop/ocaml-ctypes#41
• Static linking under OSX yallop/ocaml-ctypes#96

[aantron]

I'm currently using this, which is based on @avsm's code in the YAML bindings linked by @yallop. It probably doesn't qualify as easy, but it does allow static-only linking, and the resulting binary does not need libffi.

[mbacarella]

@craigfe has a good minimal dune example of binding a library where you have to include a header here
https://github.com/CraigFe/ocaml-fiu/blob/main/src/fiu/ffi/dune

[ghost]

FTR, ctypes support in dune is scheduled (though not started yet). @voodoos will be working on it.

[mbacarella]

Let me know if I can help. For reference, it takes ~60 lines of boilerplate-ish dune to generate c-stubs for a library with a header, constants and functions:
https://github.com/mbacarella/mpg123/blob/main/c/dune

[rgrinberg]

@mbacarella some help would definitely be appreciated. If you'd like to contribute, we can guide you through the process.

As a first step, what kind of ctypes stanza do you propose to wrap all that boilerplate?

[mbacarella]

Suppose we wanted to write a stanza for binding for the mpg123 library (linked above).

I suggest that because it's a fairly complete example, since it involves

• An easily installed but not necessarily present third party library that will end up in a standard enough place
• Generates stubs for constants and C-wrapped types, as well as function stubs, which requires extra inclusion of C headers
• Uses pkg-config to get cflags and lib invocation flags (new as of today!)

The minimum possible stanza I can think of would be this:

(ctype_library
  (name mpg123)
  (pkg_name libmpg123)
  (c_headers "#include <mpg123.h>")
  (type_descriptions Mpg123_c_type_descriptions)
  (function_descriptions Mpg123_c_function_descriptions))

The user should then provide these two files.

mpg123_c_type_descriptions.ml

module Types (F : Ctypes.TYPE) = struct
  open Ctypes
  open F
  let mpg123_api_version = constant "MPG123_API_VERSION" int
  let mpg123_ok = constant "MPG123_OK" int
  let mpg123_done = constant "MPG123_DONE" int
end

mpg123_c_function_descriptions.ml

open Ctypes

module Types = Mpg123_c_types

module Functions (F : Ctypes.FOREIGN) = struct
  open F
  let mpg123_init = foreign "mpg123_init" (void @-> returning int)
  let mpg123_exit = foreign "mpg123_exit" (void @-> returning void)
end

Example usage

The user is then free to use this ctype library in the rest of their ocaml project by listing mpg123 in their libraries and accessing via:

module Types = Mpg123.Mpg123_c_types
module Functions =
  Mpg123_c_function_descriptions.Functions (Mpg123.Mpg123_c_generated_functions)

I have to say, this seems pretty cumbersome and confusing to the end user. Adding a ctype_library stanza may be a win because it makes a really cumbersome confusing hard thing possible, though I could see the user being utterly mystified by being asked to include names of modules/libraries that don't otherwise appear in their tree and if the generated boilerplate is wrong the user might be even more lost in the weeds trying to make sense of it.

What do y'all think?

[rgrinberg]

That's a good start. Perhaps a field inside the library stanza is possible as well?

(library
 (c_types ..))

With similar fields inside such as yours.

though I could see the user being utterly mystified by being asked to include names of modules/libraries that don't otherwise appear in their tree and if the generated boilerplate is wrong the user might be even more lost in the weeds trying to make sense of it.

I agree that this is problematic. We should always strive to avoid magic names and ask the user to provide all binders.
Perhaps like this:

;; inside a library stanza or on its own
(c_types
 (generated_modules
  (mpg123_c_types -> (types mpg123_c_type_descriptions))
  (mpg123_c_generated_functions -> (functions Mpg123_c_function_descriptions.Functions))))

[mbacarella]

Saying c_types as a sub-stanza in a library sounds fine as well.

;; inside a library stanza or on its own
(c_types
 (generated_modules
  (mpg123_c_types -> (types mpg123_c_type_descriptions))
  (mpg123_c_generated_functions -> (functions Mpg123_c_function_descriptions.Functions))))

I like the general idea here, but not sure what the -> would mean. That is, why not merely (mpg123_c_types (types mpg123_c_type_descriptions))?

[rgrinberg]

In select dependencies, we use the -> syntax. Personally, I just like this syntax when creating associative data. It's definitely optional though.

[mbacarella]

I'm happy to keep it, it reminds me of matching, which is comforting.

[rgrinberg]

Anyway, the syntax doesn't matter. This seems like a good starting point to me and you can implement the stanza parsing by looking at dune_file.ml. In particular, see how we handle menhir/coq stanzas. Ctypes will need something similar in the dune-project file:

(using ctypes 0.1)

To enable the feature. We aren't going to commit to stability in the beginning.

Once you make a WIP PR, I can guide you what else needs to be modified and explain the basics of dune's source.

[mbacarella]

Sounds good, let me take a crack at this.

[avsm]

Great to see you working on this. I’d be happy to print ocaml-yaml over to this when there’s a PR to try out.

…

On 28 Oct 2020, at 16:38, Michael Bacarella ***@***.***> wrote:  Sounds good, let me take a crack at this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[mbacarella]

Great to see you working on this. I’d be happy to print ocaml-yaml over to this when there’s a PR to try out.

Sorry to steal your thunder but thought I'd try porting ocaml-yaml over to it to help debug the feature and get it ready.
avsm/ocaml-yaml#41

[mbacarella]

Also ported luv over to this as well:
aantron/luv#114

This feature is being reviewed now, targeting landing in dune 3.0.

[rgrinberg]

Fixed by #3905