Escaping Function Pointers from Sulong to Native Code

[mrigger]

It can happen that programs pass Sulong function pointers to a native function. Since a Sulong function pointer is not a function that can be called by the native side, we currently throw an error and exit the program.

Passing a Sulong function pointer to the native side is not uncommon, since the C standard library has functions that expect function pointers that are usually implemented by the user, e.g., the qsort function.

[mrigger]

In the long term, we want to compile the standard libraries to bitcode to prevent this case. As a short term solution we can provide implementations of the qsort and other functions in a C file, which is then compiled and linked by Sulong as a "partial" standard library.

[pekd]

But even then, how will this handle external libraries? Would you also compile them to bitcode? What about libraries without source code? Decompile them to bitcode (question is: does this always work? and this would somehow defeat the purpose of "shared libraries", because then you will have multiple copies of the same "shared library" in memory)?

Should Sulong ever work on Windows, how would you register callbacks via the Win32 API then? Because there you cannot even decompile anything to obtain bitcode. And probably there are similar cases on Linux too. Or is the solution to that something like "don't use Sulong in such cases"?

I don't think, that just compiling the standard library to bitcode will solve this problem; it will solve it just in a few cases, and even then you will have the same problems as with statically linked binaries: if there are bugfixes/security fixes to such libraries, you have to apply them independently of the rest of the system.

[mrigger]

I describe the approach we want to use for external libraries in a ECOOP doctoral symposium paper (mainly section 3.3). We indeed want to compile them to bitcode if the original source code is available, and otherwise use binary translation to "decompile" them to bitcode. I doubt that the existing binary translation approaches are mature enough to decompile everything. There is still some work ahead of us. ;)

I did not think about Windows yet. My focus is to make a proof of concept that shows that it is possible under Linux.

I do not understand your point about statically linked binaries. We compile a shared library once to bitcode, and all programs that rely on this library can include it. We do not statically link the main program with the shared library bitcode files. Thus, bugfixes or security fixes only need to be applied once.

[pekd]

The point I was trying to make is: do you package the bitcode files of shared libraries with Sulong, or do you decompile the system's libraries once at every startup? If you package them with Sulong, you have the same situation as on Windows, where every program brings its own version of a DLL (or on Linux, if a program uses a statically linked library): if such a library has to be updated, it is no longer sufficient to update the system's library via the regular update mechanisms, but instead you will have to provide an update to Sulong's version as well.

[chrisseaton]

I think answers to these questions are highly application specific. What would be right for running Ruby C extensions would be very different for running a standalone C application. So I don't think there are broad answers and we will need to support lots of different options.

[mrigger]

We will download and compile the standard libraries when Sulong is used the first time. Then, all the applications run by Sulong will use them. That is different to the situation on Windows and its DLLs since the "Sulong standard libraries" are shared between the user applications and exist only once on the hard disk. But as you said, we will have to update the standard libraries ourselves and cannot simply use the system's regular update mechanisms. I do not think that this is a concern, since we could also offer our Sulong standard libraries as packages that can be downloaded by the system's package manager (at least on Linux).

[grimmerm]

This is no problem anymore with Truffle Native Interfrace.