Wrapping `nfloat` in Arblib.jl

[Joel-Dahne]

I started an attempt of wrapping the new nfloat module in Arblib.jl. I thought I would create this issue to collect some of the issues and decisions encountered along the way. I'm creating the issue here on the Flint page instead of the Arblib.jl page since the discussions could be relevant for Flint itself.

It is relevant to know that the way Arblib.jl works is that it parses the Flint documentation and automatically generates the low level wrapping of the functions. In particular this means that having enough information in the function signatures to automatically generate the wrappers is important, otherwise a lot of manual work is required.

So far I have two comments:

1. The functions nfloat_ctx_set_real_prec and nfloat_ctx_set_real_prec have different names in the documentation and the code. In the code they have a leading underscore. I'm not sure which one is the intended one?
2. Many Flint types (such as arb) have a _t-suffix version that is used for scalar values and the _ptr and _srcptr-suffix version that is used for vectors of that type (e.g. arb_t, arb_ptr and arb_srcptr). In Arblib.jl the _t version is mapped to the scalar type and the _ptr and _srcptr versions are mapped to vectors of that type (arb_t is mapped to Arb, and arb_ptr and arb_srcptr are mapped to ArbVector). I don't think this approach is used in all of Flint, but at least for the types coming from Arb it is mostly followed. For nfloat this approach is not followed, it uses nfloat_ptr and nfloat_srcptr for everything. This is problematic for the automatic wrapping because we don't have enough information to determine if the argument is a scalar or a vector. My understanding is that the reason for this is that nfloat is not actually a single type, but each precision is technically its own type, is that correct? One possible solution to this issue (from the point of view of Arblib.jl) could maybe be to define typedef void * nfloat_t (which is exactly the same definition as for nfloat_ptr) and use that type whenever a scalar value is intended. Would this be technically possible? I'm however not so fond of forcing Flint to structure it's code in a certain way just to make the life for Arblib.jl easier (though of course it could also help any other attempt of automatically generating wrappers). In this case it could however maybe be beneficial in terms of the Flint documentation as well, the type signature nfloat_ptr doesn't tell the user if the function expects a scalar value or a vector so using nfloat_t for scalar values could be helpful for that reason. The could be some other reason for the choice to use nfloat_ptr instead of nfloat_t that I'm missing though?

If I encounter any more issues in my wrapping attempt I'll update you!

[fredrik-johansson]

1. That's a bug; I guess they shouldn't have the underscore.

2. There is no nfloat_t for the same reason that there is no gr_t: you couldn't declare a stack variable of that type in C. We do define nfloat64_t, nfloat128_t, nfloat256_t, ..., and one option would be to wrap each of those. But I guess you want to expose something with the precision as a parameter as well. To some extent the lack of distinction between single arguments and vectors of arguments is by design, but I'll think more about that...

[Joel-Dahne]

I don't think there should be any issue with actually wrapping the underlying types. For nfloat my current approach is

mutable struct nfloat_struct{P,F}
    head::NTuple{NFLOAT_HEADER_LIMBS,UInt}
    d::NTuple{P,UInt} # FIXME: Should be different for 32 bit systems
end

Where the P::Int parameter determines the precision (the precision is 64P) and the F::Cint parameter determines the flags to use. This way we can make sure at the type level that we don't mix values with different types or different flags (I'm not entirely sure if this is the best way to handle the flags. In C it is technically possible to use a (say) nfloat64_t value with different ctx as long as they have the same precision (but the flags could be different). But I'm not sure to what extent this should be allowed in the Julia interface.)

The issue with nfloat_ptr is rather with parsing the documentation for generating the wrapper functions. To elaborate a bit more on this take for example the functions

int nfloat_add(nfloat_ptr res, nfloat_srcptr x, nfloat_srcptr y, gr_ctx_t ctx)
int _nfloat_vec_add(nfloat_ptr res, nfloat_srcptr x, nfloat_srcptr y, slong len, gr_ctx_t ctx)

and lets say we have the (higher level) types NFloat and NFloatVector (this is how the code is currently structured for arb and acb). For the nfloat_add function we then want nfloat_ptr and nfloat_srcptr to be mapped to NFloat for the wrapper. Whereas in nfloat_vec_add we want them to be mapped to NFloatVector. Currently there is no way for our wrapper to differentiate between these two cases, as it currently stands we would either have to use some heuristics for deciding when something is a scalar or a vector (such as having vec in the function name, or be proceeded by a len argument), or we would have to hard code a lot of special cases. For arb (and acb) this problem doesn't arise because we always map arb_t to scalar types and arb_ptr and arb_srcptr to vector types.

What would be useful for the wrapper is to have some distinction between scalars and vectors in the type names in C, even if all of them are just typedefs for void pointers in the end. In some sense similar to how Flint differentiates between gr_ptr and nfloat_ptr, even though they are both just typedefs to void pointers. Of course making such a change might not be in the interest of Flint, I don't even now what it would amount to. In that case we should still be able to make a Arblib.jl wrapper for nfloat, it would just require a bit more manual work. It might not be to bad in the end, the nfloat module has a somewhat limited amount of functions.

How is it for the gr module? It has the type gr_vec_t that is used for vectors, so we would be able to differentiate between scalars (gr_ptr and gr_srcptr) and vectors (gr_vec_t). There is some functions in gr_generic which use gr_ptr for representing vectors, but there is a TODO about moving these functions to gr_vec so I'm not sure if these will stay? Is there some other place where gr_ptr is used for vectors? I couldn't immediately find any (but the documentation is also a bit sparse for some of the generic code).

Note that our current approach already has some issues related the interpretation of pointers, but it only occurs in a few places for now. For example the wrapper of the function

slong acb_theta_ql_reduce(acb_ptr new_z, acb_t c, arb_t u, slong * n1, acb_srcptr z, const acb_mat_t tau, slong prec)

maps the slong * n1 argument to Vector{Int}, even if it would be more appropriate to map it to Ref{Int}. Most instances of slong * (at least in the code coming from Arb) does however refer to a vector of slong rather than a single one. (I just realized this precise function is not actually wrapped in Arblib.jl due to missing to include the corresponding file...).

[fredrik-johansson]

Note that gr_vec_t is a memory-managed type, similar to gr_poly_t.

We still use gr_ptr / gr_srcptr for "raw" vectors.

[Joel-Dahne]

Somehow I missed that gr_vec makes heavy use of gr_ptr as well. There is indeed then quite a lot of cases in the generic interface when it is not possible to know from just the function signature if the arguments should be interpreted as scalars or vectors. Since the generic interface has such a large number of methods that would make an automatic wrapping using our current approach in Arblib.jl fairly difficult.

For nfloat I think it should be fairly straight forward as it is. As far as I can tell the only cases where nfloat_ptr should be interpreted as a vector is for the methods with vec in their name. We could then on our end manually add a dummy version of these along the lines of

int _nfloat_vec_mul_scalar(nfloat_vec_ptr res, nfloat_vec_srcptr x, slong len, nfloat_srcptr y, gr_ctx_t ctx)

where we use nfloat_vec_ptr and nfloat_vec_srcptr as an internal name for arguments that should be vectors. This would add a little bit of manual work on top of the automatic parsing of the documentation, but since it is a fairly limited number of functions it should still be fine.

[Joel-Dahne]

I have create a proof of concept PR over at Arblib.jl

[Joel-Dahne]

In the end the only two issues I encountered was the wrong name for the real_prec functions and the issue with distinguishing scalars and vectors. Since the name has been fixed and distinguishing between scalars and vectors will be handled on the Arblib.jl side I think it is fine to close this!