selectively exposing C functions from header file

[hros]

I am upgrading a package to use CBinding's automatic parsing of C header files (previously I performed this manually using ctypedef to define structures and writing inline julia functions to perform ccall to the C functions).
The structs are imported perfectly (almost magically ;) ), and I am seeing for each of the C functions two new elements in the module:

• c"function_name" - seemds to be undefined when checking its type (although it is listed as part of the module)
• Cbinding_function_name of type UnionAll

1. Are functions which are defined in the C header imported by Cbinding as Julia functions?
   If so, please link to a usage example?
2. Please explain the meaning of c"function_name", Cbinding_function_name
3. Can I control what elements of the C header are imported into Julia by Cbinding?
4. I am getting multiple warnings Failed to find <C symbol> in: or the Julia process. What do they mean

thanks

[krrutkow]

A C function is bound to Julia as a singleton type (Cbinding_function_name) and an instance of that type (c"function_name") for various reasons. There is also a function ((::Cbinding_function_name)(...) = funccall(...)) defined which does the necessary work to call the C function, like converting types, properly dealing with variadic function arguments, etc. These are implementation details that will likely change in the future.

Unfortunately, there are often header files containing prototypes to functions which do not actually exist in the library (or defined in a sneaky manner, like GLIBC sometimes does). So in essence, the header file is lying and claiming the library will contain something it doesn't, but from our perspective we do not know if it is the header file that is wrong or the library (or perhaps a library we are missing). So the warning is the indication that such a scenario exists.

It is not possible to selectively import symbols, but if the warnings are fine to ignore, then you can add the q string macro option to suppress those messages.

[hros]

@krrutkow, thank you so much for your great work on the package and the "hand holding" of developers

I will ignore the warnings.
As for the generated functions, I currently have wrappers for the relevant C functions, but I would like to try and use Cbinding's generated functions.
I can't seem to call them
Do you have an example you can share that demonstrates this?
I tried checking the parameters of the julia generated function with methods(cbinding_func_name).
methods(Cbinding_func_name) returns

# 0 methods for type constructor

and methods(c"func_name") returns two methods:

1. `(func::module_name.Cbinding_func_name)(var"c"arg1"", var"c"arg2"")
2. (cb::CBinding.Cbinding{<:CBinding.Cfunction})(args...)

[krrutkow]

The README gives some examples of calling the functions. Since your c"func_name" appears to take 2 arguments (not knowing the types without more details), you should be able to call it with something like c"func_name"(123, 456). What issue are you encountering trying to call them?

[hros]

The functions have pointers to data as arguments
In my wrapoed functions I used Julia's Ptr and Ref
Do you have an example of a function that takes a pointer?

[krrutkow]

Depends on how the function is using the pointer, but something like this should be possible.

using CBinding
c``
c"""
  struct S {
    int i;
    int j;
  };

  inline static void func_name(struct S *a, struct S *b) { *a = *b; };
"""w

ptr = Libc.malloc(c"struct S")  # malloc an instance (uninitialized)
ref = Ref(c"struct S"())  # create a zero-ed instanced
@info ptr[]  # [ Info: var"(c\"struct S\")"(i=65713851, j=0)
c"func_name"(ptr, ref)
@info ptr[]  # [ Info: var"(c\"struct S\")"(i=0, j=0)

[hros]

Thanks
While your demo code works, when I try the code from the package I am working it - it does not work.
Could you have a look at the XXhash project, specifically the header file which has full implementations of all the library function
There is a jll package of the library, which you can use with using xxHash_jll
try calling the XXH32_createState() which does not take any input and returns a pointer to the type c"XXH32_state_t" (for which the Julia equivalent can be defined with: const XXH32_state_t = c"XXH32_state_t")
When I call the function, I get the following:

julia> c"XXH32_createState"()
ERROR: could not load symbol "XXH32_createState":
julia: undefined symbol: XXH32_createState

[krrutkow]

Hmm, it works for me...

julia> module libxxhash
       using CBinding
       using xxHash_jll
       c`-I $(xxHash_jll.artifact_dir)/include -L $(xxHash_jll.artifact_dir)/lib -lxxhash`
       const c"uint64_t" = UInt64
       const c"uint32_t" = UInt32
       const c"size_t" = Csize_t
       c"""
         #include <xxhash.h>
       """j
       end
Main.libxxhash

julia> state = libxxhash.XXH32_createState()
CBinding.Cptr{Main.libxxhash.var"c\"struct XXH32_state_s\""}(0x0000000005902f00)

[hros]

First, thanks.
Now, I am confused....

I see that you are not using the c"XXH32_createState". I thought that this is the generated function that should be called (following your previous example: c"func_name"(ptr, ref))

When I create the Julia wrapper functions, I am using ccall calling the (highly) optimized implementation from xxHash_jll.
I thought that CBinding can capture the function name, return value and arguments and create a wrapper that calls the library.
Does the usage of c`...-lxxhash` mean that a new function is compiled and linked by CBinding?
Is libxxhash.XXH32_createState() calling the function in the library (the xxHash_jll) or a newly compiled and linked function?

If it is the latter (not using the functions in the jll package), then I will stick to manual wrapping functions.
Also, I might as well copy the data structures to a c""" .... """ block instead of including the header file.
The downside is that I still must manually follow changes in the original C library, I thought I could get away with CBinding taking care of automatic parsing of all functions

[krrutkow]

I see that you are not using the c"XXH32_createState"

Because I used the j string macro option it also generates Julia-native names for the bindings, so I did use it but not explicitly.

julia> using .libxxhash

julia> libxxhash.XXH32_createState === c"XXH32_createState"
true

I thought that CBinding can capture the function name, return value and arguments and create a wrapper that calls the library.

This is exactly what is happening, but with a level of indirection in place. The function signature (captured in the Cfunction{...} type), the library, and symbol are wrapped in the Cbinding below...

julia> supertype(typeof(c"XXH32_createState"))
Cbinding{Cfunction{Cptr{var"c\"struct XXH32_state_s\""}, Tuple{}, :cdecl}, Symbol("/home/user/.julia/artifacts/13ab51aa94fca17cd1fbd3b9ce14f303e481893e/lib/libxxhash"), :XXH32_createState}

So when calling c"XXH32_createState"(), the library, symbol, and function signature are all used to make a ccall within the CBinding code.

Does the usage of c`...-lxxhash` mean...

That tells CBinding what libraries to look in (and also where to look for headers/libraries) for the bindings to find their symbols in and bind to, as shown above.

I thought I could get away with CBinding taking care of automatic parsing of all functions

I'm not sure what you mean by "automatic parsing", but CBinding does automatically generate all of the types and function bindings for the header and library in xxHash_jll. If using something like the libxxhash module definition I had posted above, then there shouldn't be anything left to do other than to use it similar to how you would normally use xxHash from C code. No need to use ccall or anything since calling the bindings in the module will call functions in the library in xxHash_jll.

[hros]

Brilliant
I'm dropping my julia-wrappers file
Thanks
btw, a few more examples would be extremely helpful. especially, relating to the string suffix options for the c block

[hros]

To quote Steve Jobs: "one more thing"
Some of the C functions return an enum value:

typedef enum {
    XXH_OK = 0, /*!< OK */
    XXH_ERROR   /*!< Error */
} XXH_errorcode;

When calling the functions in julia, I see the following return value:

var"(c\"XXH_errorcode\")" (0x00000000)
->c"XXH_OK"    (0x00000000)
  c"XXH_ERROR" (0x00000001)

I can compare the result to either c"XXH_OK" or 0
Checking the type of the result in julia:

var"(c\"XXH_errorcode\")" (2 options)
  c"XXH_OK"    (0x00000000)
  c"XXH_ERROR" (0x00000001)

I understand that the var"..." syntax is for mangled C names.
Please explain the var"..." type and return value in this example

Also looking at methods of functions, for example the following C function:

XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length)

I see the following method prototype:

# 2 methods for callable object:
[1] (func::Cbinding_XXH32_update)(var"c\"statePtr\"", var"c\"input\"", var"c\"length\"") in Main at ...
[2] (cb::Cbinding{<:Cfunction})(args...) in CBinding at ...

why does methods return the argument names and not their types (as is the case with other Julia functions)

[krrutkow]

Please explain the var"..." type and return value in this example

Because Julia lacks support for circular type references (JuliaLang/julia#269), and that is encountered a lot in C type definitions, the generic approach taken by CBinding is to do some sneaky maneuvering using abstract types, which are actually named to match the C types (var"c\"XXH_errorcode\""), with a single bits type as a subtype, which are named with the parentheses added (var"(c\"XXH_errorcode\")"). Except for the occasional use of bitstype(...), you would normally never use or need to know about the abstract vs. bits types detail. The normal usage, like c"XXH_errorcode", is actually using the string macro to expand to the var"c\"XHH_errorcode\"" mangled name.

All the trickery will go away someday when Julia has incomplete type. This section of the README might offer some additional information too.

why does methods return the argument names and not their types (as is the case with other Julia functions)

Well, technically the argument types are all Any, and since the function is just a sophisticated wrapper around ccall, automatic type conversions will be handled with Base.cconvert. I have considered restricting the argument types for the binding functions (e.g. using Signed/Unsigned/Number/etc. as the types instead), but restricting the types any more would make usage of the binding functions more tedious as the caller of the bindings would need to handle all of the type conversions instead.

If you just want to reference the C argument/return types for a binding, you can use CBinding.argtypes(c"func_name") and CBinding.rettype(c"func_name"), or you could reference the help for the binding with ? libxxhash.XXH32_createState or ? var"c\"XXH32_createState\"".

help?> libxxhash.XXH32_createState
  XXH32_state_t *XXH32_createState()

  Defined at xxhash.h:481 (file:///home/user/.julia/artifacts/13ab51aa94fca17cd1fbd3b9ce14f303e481893e/include/xxhash.h)

  Allocates an XXH32_state_t.

  Details
  =========

  Must be freed with XXH32_freeState().

  Returns
  =========

  An allocated XXH32_state_t on success, `NULL` on failure.