Should Chapel support FCFs implemented as function pointers? If so, how?

[bradcray]

In discussions about first-class functions (FCFs) and closures, we sometimes talk about
there being two weights, where both weights tend to involve an object (class or record)
of some kind. I often find myself wondering whether we could/should support a
simple C-style function pointer variant as well because many of the cases where
I think of wanting to use a FCF seem as though that's all would be needed (i.e.,
"couldn't we have this fairly easily today?"). And in C interop scenarios, they can
obviously be useful (to pass out to C; or even receive from C?).

If we were to support such function-pointer style FCFs in Chapel, is this something
that would be visible to the user in the syntax of the language, or would it be more
like an optimization to the more general FCF support that we have?

I'm opening this issue to see whether people tell me "we really don't need that because..."
or "yes, we should have this and I always assumed we would" or whatever, and to serve
as a place for collecting those thoughts (because I worry I bring it up redundantly all
the time in 1:1 conversations and the like).

[twesterhout]

A random thought / note. libffcall (https://www.gnu.org/software/libffcall/) is an interesting pure C library that may be used to create proper closures that look like function pointers to the outside world. Perhaps, this could be a quick way of adding support for closures to the C backend...?

[bradcray]

Tagging @dlongnecke-cray on this since it seems as though Tom's comment may be of interest to him if it's not something he's already investigated.

[dlongnecke-cray]

Thanks for the great suggestion, Tom! Awhile back Michael and I discussed what the closure representation might end up looking like under the hood, and as I recollect, we both seemed to like the idea of it being a function pointer that takes a void* environment pointer as a first argument. In practice with how these C variadic closures are created and desugared, I expect the two will probably end up looking very similar in the end.

I'm not particularly concerned or worried about special handling for the C backend (vs say LLVM). I think the lowering on our end will all occur before that point, and it will just be business as usual for either backend.

---

Some thoughts on function pointers, based on my last work:

Function types themselves are lowered several times even before code generation. One example is during LowerErrorHandling. The signatures for throwing functions must be lowered to remove the throws tag and accept an Error as the first argument. This is not conceptually difficult - it's mainly a matter of finding and documenting all the spots where we implicitly mutate the type of a function. In many ways the new frontend will make this process easier, as we can decide on and lower to a single normal form at one point instead of all over the place as we do now.

The tricky part about getting function pointers to work in a multi-locale environment is extending the global function tables that the runtime and compiler collectively maintain to work with this more general notion of a function pointer.

We already build up these tables with the addresses of on statements during compilation, so it's mainly a matter of adjusting the codes and typing in the generator so that the tables accept function pointers more generally.

Once we've done this, my expectation is that a closure (and its type) won't appear wildly different than other function types maintained in this global table - it's mainly the process of calling it and packing up all its state that will differ. But that should all be handled earlier during compilation. Down at codegen it should just look like any other function pointer call.

[twesterhout]

@dlongnecke-cray thanks for the explanation! This seems very well thought through. The only question/concern that I have as a user (rather than the person doing all the work hehe) is with the void* argument. Would that void* argument be visible to the user if they choose to pass the closure to an external library? It'd be a bit cumbersome if one had to create a closure in Chapel, pass the c_fn_ptr and the void* parts to the external code separately, and then have the external code remember that void* needs to come first instead of last etc. That's just a thought though, and I'm likely missing some context that'd render my argument moot 😄

[dlongnecke-cray]

Yeah, one advantage that the varargs has over our approach is that you can implicitly shove all the outer variables onto the stack frame without needing the void*. We haven't really put much thought into passing closures to external functions (whereas, it is definitely our intention to let you pass regular old function pointers back and forth between say C and Chapel).

Do you imagine that passing closures to external routines will be important or common for you? It has implications for the lifetime/management of outer variables as well as how we potentially type a closure. One part of that being the extern signature (as you bring up).