ccall should automatically convert integers

[JeffBezanson]

This can be done for arguments whose types can be inferred. It will make things more convenient in general, but the cost is that it won't be easy to predict when an explicit conversion needs to be written. At the very least it will improve the situation with literals, so you don't need to write int32(0).

[StefanKarpinski]

This makes me think: can we use type inference with numeric literals? This is sort of the opposite of what other languages do, but it would probably give us 99% of what we want: when a numeric literal is used in a context where we can figure out what type it ought to be — e.g. x + 0 — can we just make it be of the right type automatically? When inference is impossible, numeric literals are just of the default type for that kind of literal. Sure, it doesn't work if you have a function or something, but whatever. It handles most of the cases and requires writing zero(x) and one(x) a million times fewer.

[StefanKarpinski]

In fact, maybe this is just specific case of that — that would make it less confusing: variables have to be cast, literals will automatically be converted to the right type.

[JeffBezanson]

Actually that is exactly what Haskell does. + is defined only for arguments of the same type, so one argument can determine the type of the operation and force a literal to be of the same type. I don't see how we can do that though. Both arguments determine what method to apply, and any method signature can be defined, so there isn't a unique answer to what type an argument "ought" to be.

[StefanKarpinski]

Back to the original issue: can we have ccall always convert(T,v) before applying? That would make the way ccall works a lot simpler and more transparent.

[JeffBezanson]

The big problem here is the way we handle pointers. If you pass an Int32 to a ccall expecting Ptr{Int32}, it passes a pointer to your integer value. But converting an integer to a pointer uses the bits of the integer as the value of the pointer.

Another issue is when to insert the calls to convert. Since ccall uses normal function call syntax, it's hard for the front end to know exactly what is and isn't a ccall. I guess we could make ccall a reserved word.

[StefanKarpinski]

The big problem here is the way we handle pointers. If you pass an Int32 to a ccall expecting Ptr{Int32}, it passes a pointer to your integer value. But converting an integer to a pointer uses the bits of the integer as the value of the pointer.

See, I didn't know that. ccall is kind of a big mystery and has all sorts of magical behaviors. It would be nice to make it less magical. I think it's acceptable to require calling pointerto(1) if you want a pointer to an integer. You have to do one = 1; and later use &1 in C, so this is not too difficult. Maybe we want a different function than just convert for this — cconvert or something like that — so that it can have different behavior than normal convert.

Another issue is when to insert the calls to convert. Since ccall uses normal function call syntax, it's hard for the front end to know exactly what is and isn't a ccall. I guess we could make ccall a reserved word.

I would be completely find with ccall being a reserved word. It's a pretty key piece of infrastructure and its not like it's actually reasonable for anyone to add methods to it or redefine it or anything like that.

[JeffBezanson]

If a garbage collection occurs between calling pointerto() and the ccall that uses it, you could get a segfault. There's a reason for the magic. Actually, exec in process.j seems to have this bug already.
It seems we would need to expose the needed object pinning to julia, like

@pin begin
    ...
    p = pointerto(pin(x))
    ...
    ccall(...)
end

[StefanKarpinski]

Hmm. That's nasty. How about preventing GC for the duration of the function instead?

[JeffBezanson]

Preventing GC during the ccall doesn't help, since the problem is GC between pointerto and ccall. Preventing GC during the function that uses ccall is a complete non-starter.

[StefanKarpinski]

How about this: pointerto doesn't actually create a pointer to an object immediately, but rather just wraps it to indicate to ccall that it should take a pointer to the object in question, rather than applying convert to it as it normally would. In fact, in this case, we should just write pointerto(obj) as Ref(obj) — since that already essentially does that. Then we just need to make the conversion of a Ref to a C pointer value be a pointer to its referent, which seems sane. Since the actual dereference is under ccall's control, GC can be stopped before it happens. Does this make sense?

[JeffBezanson]

I think that would do it, since we need some way to tag that a value needs to be converted differently. Doing heap allocation for this is unfortunate though. Actually you can already do this by wrapping numbers in arrays, x => [x].

I can rewrite the ccall to keep the original arguments so the code generator can make sure they are gc roots:

ccall(:f, R, (A, B), arg1, arg2)

to

ccall(:f, R, (A, B), convert(A,arg1), convert(B,arg2), arg1, arg2)

So there is no need to disable GC.

[StefanKarpinski]

Do you mean this:

ccall(:f, R, (A, B), convert(A,arg1), convert(B,arg2))

? Any chance that the compiler can figure out that heap allocation is unnecessary because the Ref object doesn't escape the function? This, of course, requires a special understanding that passing something to ccall doesn't actually cause it to escape, while at the same time causing it to become a GC root during the execution of the ccall.

[JeffBezanson]

Well, that's probably possible with enough of my time :)

I think the bigger problem is that a lapack call would go from this:

int32(m), int32(n), QR, int32(m), jpvt, tau, work, lwork, info

to this:

Ref(m), Ref(n), QR, Ref(m), jpvt, tau, work, Ref(lwork), info

which is not the kind of improvement I was hoping for by filing this issue :)

[StefanKarpinski]

But the LAPACK calling convention is clearly not the norm — it's very ugly. Optimizing for it seems silly. I think that the latter version is actually much simpler to understand: it's shorter and the fields that should be passed as pointers are instead passed as Refs. That's a pretty simple thing for users to wrap their heads around:

• Convert is called to convert each argument to the type expected by the function
• Refs are used for values passed by reference (i.e. pointer)

This would be infinitely more explainable and usable than what we currently have, IMO.