RFC: ccall with julia types as structs

[vtjnash]

This adds syntax and support for passing julia types to ccall functions, either as StructTypes (by-value) or Pointers to StructTypes (by-reference).

Some example/test code:

ccall((:cptest,"ztest"), Void, (StructPtr{ComplexPair{Int}},), ComplexPair(122,3))
ccall((:cptest,"ztest"), Void, (StructPtr{ComplexPair{Int}},), &ComplexPair(122,3))
ccall((:cgptest,"ztest"), Void, (StructPtr{ComplexPair{Float64}},), &ComplexPair(2.,3))

ccall((:cgtest,"ztest"), Void, (Struct{ComplexPair{Float64}},), ComplexPair(2.,3))
ccall((:ctest,"ztest"), Void, (Struct{ComplexPair{Int}},), ComplexPair(2,3))

ccall((:sptest,"ztest"), Void, (Struct{ASCIIString},), "asdf")

compile the following library with gcc ztest.c -shared -o ztest.so -g -Wall or gcc ztest.c -shared -o ztest.dylib -g -Wall depending on your platform:

#include <stdio.h>
#include <complex.h>
#include <stdint.h>
#include <inttypes.h>
typedef struct {
    long real;
    long imag;
} complex_t;
void ctest(complex_t a) {
    //Unpack a ComplexPair{Int} struct
    printf("%ld + %ld i\n", a.real, a.imag);
    return;
}
void cgtest(complex double a) {
    //Unpack a ComplexPair{Float64} struct
    printf("%g + %g i\n", creal(a), cimag(a));
    return;
}
void cgptest(complex double *a) {
    //Unpack a ComplexPair{Float64} struct
    printf("%g + %g i\n", creal(*a), cimag(*a));
    return;
}
void cptest(complex_t *a) {
    //Unpack a ComplexPair{Int} struct pointer
    printf("%ld + %ld i\n", a->real, a->imag);
    return;
}
void stest(char *x) {
    //Print an Array
    printf("%s\n", x);
}
void sptest(struct { struct { void* t; char* x } *x } x ) {
    //Unpack an ASCIIString
    printf("%s\n", x.x->x);
}

[pao]

It will be nice to have this built into ccall().

[ViralBShah]

I presume this will make it possible to address #85.

[vtjnash]

For x86, yes this works for #85 since a ComplexPair{Float64} has the same struct layout as a C99 complex float. For other architectures, that may not be true. Note that Complex128 is still useful for storing a complex number inline or in an array (again, for x86). I'm not certain how to address that distinction without being able to have Struct be a bitstype with a variable number of bits (depending on the parameter)

It should be relatively straightforward to implement this for return types also (and may even partially work already, although I haven't tested it). I also should be able to also provide some field accessors for StructPtr types, similar to how unsafe_ref / unsafe_assign work for pointers.

[stevengj]

@vtjnash, C99 (and Fortran, and C++) require complex numbers to be stored as a consecutive pair (real,imag) of floats, so the struct layout should not be a problem. As I understand it, the only question is what the calling convention of the ABI is for complex arguments (and return values) passed by value.

@JeffBezanson, does this mean Complex64 etcetera can go away (or become an alias) in favor of ComplexPair{Float32}, or should that wait on immutable types? If Complex64 does not go away, presumably the ccall interface should translate this into a Struct?

[JeffBezanson]

Using an llvm struct type should be fine, except for complex on x86-32,
where there is no abi standard for c99 complex iirc. But I suspect there is
a reasonable de-facto standard. Gcc probably does something reasonable.
On Dec 31, 2012 11:09 AM, "Steven G. Johnson" notifications@github.com
wrote:

@vtjnash https://github.com/vtjnash, C99 (and Fortran, and C++) require
complex numbers to be stored as a consecutive pair (real,imag) of floats,
so the struct layout should not be a problem. As I understand it, the only
question is what the calling convention of the ABI is for complex (and
struct) arguments (and return values) passed by value.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1831#issuecomment-11779527.

[ViralBShah]

I would prefer to keep the names Complex64 and Complex128.

[vtjnash]

Currently, the Complex128 bitstype is necessary to cause the layout of a complex array to be inline. I'm not certain of the best way to handle this, but perhaps the cconvert should handle the conversion to a struct (or still special case the ComplexN types, to avoid a heap allocation).

If my research is correct, the appropriate statement is that the C standard says nothing about implementation of the complex type, nor do the i386 / x86-64 processor specs. It is the operating system that provides a document describing the convention that should be used to produce compatible code. It seems that common implementations treat the complex type as a struct { T real, T imag } since they are often derived from the System V spec. For linux on x86-64 this is standardized in the document System V Application Binary Interface for AMD64 at http://refspecs.linuxbase.org/elf/x86_64-abi-0.95.pdf. Mac has a different call ABI, but appears to still treat complex types as the aforementioned struct.

[stevengj]

@vtjnash, the C99 standard (section 6.2.5) does indeed mandate the storage format of complex types: "Each complex type has the same representation and alignment requirements as an array type containing exactly two elements of the corresponding real type; the first element is equal to the real part, and the second element to the imaginary part, of the complex number." (They defined it in terms of an array type rather than a struct since structs can theoretically have arbitrary padding between fields, although practice no extant system does this for a struct of two floats or doubles.)

So, passing complex numbers by reference in this format is totally portable. The only problem is passing them by value, in which case the format is determined by the ABI (an OS thing, although in practice ABIs are often defined by the CPU vendors) and not by the C standard.

(Fortran also mandates this storage format, and my understanding is that the C++ standard eventually adopted a similar proposal, described in www.open-std.org/jtc1/sc22/wg21/docs/papers/2002/n1388.pdf, and in practice all C++ implementations store complex numbers that way anyway as described in the proposal, nor have we heard of any subsequent counterexamples from FFTW users.)

[vtjnash]

@stevengj Great, thanks! That indicates this should definitely work for #85 in on all (compliant) systems. In this pull request, I provide LLVM sufficient information to properly select the appropriate pass-by-value ABI.

I found a reference that also mentioned Fortran uses this format for storage and call-by-value, but that complex-return-by-value is more complicated. Do you know anything about this?

[stevengj]

Fortran return-by-value ABI conventions are not in general compatible with C's, even for simpler types like integers. That's why in FFTW we were forced to change return values into parameters for our Fortran 77 interface; I couldn't find a way that worked with all the compilers at the time.

As a practical matter, there are fewer compilers and architectures these days than there were 15 years ago, so it may be easier to get something that works in practice (if not in theory).

If you use Fortran 200x, they provide a standard way to call and be called from C within Fortran, so for things where you are willing to modify the Fortran code you can use that. For BLAS, you could use the cblas interface.

[vtjnash]

preview of new interface, without Struct type (currently undergoing testing before I update this pull request):

# by reference
ccall((:cptest,"ztest"), Void, (Ptr{ComplexPair{Int}},), &ComplexPair(122,3))
ccall((:cgptest,"ztest"), Void, (Ptr{ComplexPair{Float64}},), &(2.0+3im))

# by value
ccall((:cgtest,"ztest"), Void, (ComplexPair{Float64},), 2.0+3im)
ccall((:ctest,"ztest"), Void, (ComplexPair{Int},), ComplexPair(2,3))
ccall((:sptest,"ztest"), Void, (ASCIIString,), "asdf") # not the same as (Any,)

[vtjnash]

the following example currently doesn't work in the previous commit (the previous examples do work), but is seems that it should be valid to pass any jl_bits_type of the correct size, unmodified?
ccall((:ctest,"ztest"), Void, (Ptr{ComplexPair{Int}},), C_NULL)

[vtjnash]

• TODO item: return types
• TODO item: why is jl_value_ptr still part of this commit?
• TODO item: destructuring pointers to structs with unsafe_ref
• TODO item: fix behavior with Singletons to raise an error

[StefanKarpinski]

This passes all tests on my machine.

[vtjnash]

@JeffBezanson ping. ready to merge this?

I also changed the behavior of unsafe_ref and unsafe_assign. They work the same as before for Ptr{ ::BitsKind }. But now, unsafe_ref interprets Ptr{ ::CompositeKind } as a struct[] reference (inline storage of an element of size N) and creates a new Julia type with the contents. unsafe_assign currently does not implement the complementary operation (it throws a compile time error). Both currently still require compile-time type info (e.g. the julia wrapper function), is that OK or should I add runtime versions too?

[JeffBezanson]

That's OK. What is the tty cleanup stuff? Should that be picked onto master separately?
Also need rebase.

[vtjnash]

It calls uv_tty_reset_mode() after most signals. It's not really necessary anymore since the repl is handled by readline (although we may want to call the readline cleanup function there instead).

I wasn't going to bother rebasing until you were ready to merge. You around / on IRC?

[JeffBezanson]

I'm on IRC.

[timholy]

HUGE!

(Just catching up on by-now "old" news...)