Introduce unified Lua ffi.cdef builder

[Gerold103]

Lua ffi is quite fast and simple mechanism allowing to call C functions directly, without using Lua C API. It is usually faster, sometimes much faster. But ffi requires to define all the C methods again in a Lua file in ffi.cdef function. It is as bad as any other code duplication - when the original API is changed, a one could forget to update Lua files and ffi.cdef arguments. It is especially painful, when ffi C definition is really big.

Interestingly, we have exactly the same issue with module.h file - it is in fact a set of copy-pasted code blocks from some header files. But it is copy-pasted automatically. CMake calls a special routine walking over specified header files and extracting code inside /** \cond public */ and /** \endcond public */ tags.

Goal of this ticket is to introduce a similar mechanism to generate ffi C definitions. API RFC:

Option 1

File box_example.h

/* \cond ffi */

int
box_example1(void);

int
box_example2(void);

int
box_example3(void);

/* \endcond ffi */

File box_example.lua

ffi = require('ffi')
ffi.cdef_file('box_example.h')

Option 2

This option allows to collect C definitions by name, from multiple files.

File box_example.h

/* \cond lua box_example */

int
box_example1(void);

int
box_example2(void);

int
box_example3(void);

/* \endcond lua box_example */

File box_another_example.h

/* \cond lua box_example */

int
box_another_example4(void);

int
box_another_example5(void);

/* \endcond lua box_example */

File box_example.lua

ffi = require('ffi')
ffi.cdef_tag('box_example')

[Gerold103]

Just to realize the problem: SWIM needs to duplicate 96 lines from two headers; schema.lua duplicates 97 lines; tuple.lua duplicates 49 lines; buffer.lua duplicates 29 lines; digest.lua duplicates 10 lines; error.lua duplicates 27 lines; fiber.lua - 8 lines; iconv.lua - 5 lines; uuid.lua - 24 lines.

[servoin]

I propose to combine the two options into:
ffi.cdef_file(files, [, tag])
Where
"files" is a list of headers in Linux-style command line expansion (using space separated names and file masks);
"tag" is an optional tag string (like in option 2).
So, for example,
ffi.cdef_file("box_example*.h", "box_example")
will look for all .h files with names starting from box_example, and include all definitions in
/* \cond ffi box_example */ blocks.

[Gerold103]

Thanks for the proposal! It is good that this thing is started.
I think that an ability to cdef multiple files at once is an overkill. And it has nothing to do with command line. It is enough to be able to include one file. Just like C style '#include'. Then you can call ffi.cdef_file several times if you need. Although in 99% you don't need.
Also I don't think we need a wildcard. That is even a bigger overkill, which adds unnecessary complexity to the implementation.
I mean I proposed it as the option 2, but now I think it might be an overkill.

[igormunkin]

Wow, the feature is one vital for both Tarantool developers and users, and I'd like to add my two cents.

At first, I guess ffi.cdef_file is not such a good name for such a great feature. Maybe ffi.include or ffi.export, what do you think?

As for me, the processing of /* \cond */ sentinels seems to be excess, since we can just separate our public and private interfaces and split them into two headers. Then you just add to your Lua sources the following line as proposed above

ffi.include('example.h')

Moreover I see another approach to differ functions to be exported via ffi.include. Considering possible FFI misusage (e.g the one leading to FFI sandwiches), we can explicitly mark FFI function signature with a special keyword (also to be provided via LuaJIT public headers). Thus only marked functions are exported to Lua space. Please consider the following

#define FFISAFE

FFISAFE int
box_example1(void);

FFISAFE int
box_example1(void);

There are only functions mentioned in the original proposal. And what about typedefs and enums declarations?

[Gerold103]

I like ffi.include. Export does not fit. We don't export. We rather import.

Talking of cond labels. The problem is that we can't really split the headers. There is many of them, and that way does not scale IMO. Also it is unavoidable that such public headers will have #include in them, what won't be parsed by ffi. So we need tags. It is easy to implement, and allows not to change any of the existing C code.

Talking of FFISAFE sounds good. Maybe even better than tags. But then you need to design how to apply them to structures, typedefs and enums. Because often we export them too. So maybe tags are better. Don't know.

What bother me more - can we do that import at runtime? And how?

[Totktonada]

We can just load all generated definitons at once at startup, before loading built-in Lua modules. This way ffi.{cdef_file,cdef_tag,import} will not be needed.

This solution looks as easiest for me and it handles the stated problem. Aside of that we don't need to design a general purpose feature that nobody actually asks.

FFISAFE proposal looks interesting, however I don't sure it actually handles FFI misuse problem: what if a function that is called from FFISAFE one is changed? It seems that this is the primary source of misusages.

Prevention of FFI misusages deserves its own issue. I guess it requires a way more work, then automatic generation of cdefs for FFI.

[servoin]

Thank you all for your comments!
But I need a clear understanding of what should I do, because there are many opinions and ideas about the implementation.
Please provide me with the following info:

1. What will be the name of proposed function?
2. What should this function do in particular?
   Thanks.

[Totktonada]

I propose to don't add any ffi.* function, but do ffi.cdef() for generated declarations under hood.

Waiting for comments from @Gerold103 and @igormunkin.

[Gerold103]

What is 'generated declarations' you are talking about? Only a few things are a part of public API in module.h. I need to be able to export internal not generated structures, functions, types. Otherwise this is useless. Moreover, public 'generated' API is fixed. It never changes, so this code does not need that feature.

Also what is 'load at startup'? You need ffi.cdef for that, there is no other way to export structures into Lua land.

Talking of the problem about 'misusage' - it has nothing to do with this ticket. The ticket is purely about simplification of ffi.cdef.

[Totktonada]

I'm not about module.h, but another generated file. It seems I should describe my proposal better.

So, I propose to generate a file that may look like so:

local ffi = require('ffi')
ffi.cdef([[
<all declarations between /* \cond ffi */ and /* \endcond ffi */ here>
]])

Then add the file name to lua_sources list in src/CMakeLists.txt and to lua_modules in src/lua/init.c. This is basically all.

[Gerold103]

Thanks for the explanation. Yeah, that looks good to me as well. I don't have a strict preference to ffi.import or your option. Both solve the problem.

[igormunkin]

@Totktonada, thanks for the proposal! Yours one looks to fit better regarding the problem to be solved. For internal usage I'm totally OK with your solution, and I totally agree that we need another issue for "FFI misusage" problem. Could you please dump the final proposal as @servoin asked before?

[servoin]

So as I currently understand, there should be some module which generates .lua source file containing ffi.cdef with definitions from some .h file using tags to include only parts of this .h file. Correct me if I'm wrong. What I don't understand for now is how this module will be called during the build process. Could you please give an example?

[Totktonada]

Could you please dump the final proposal as @servoin asked before?

#4202 (comment) is not enough? Don't know what to add here.

What I don't understand for now is how this module will be called during the build process. Could you please give an example?

In the similar way as module.h is generated. See extra/apigen and cmake/module.cmake.

[Totktonada]

BTW, should we do the same for luaL_cdef() calls?

[servoin]

Currently, there are ffi.cdefs not only in tarantool, but also in its submodules, namely: msgpuck and small. The question is: what should we do with these ffi.cdefs in submodules? Leave them as is or apply changes for unified cdef builder (it will require separate patches for submodules)?

Also, the question about luaL_cdef() calls is still open.

[olegrok]

Leave them as is

IMO, yes, because this libraries are standalone and tarantool depends on them but not the opposite

[Totktonada]

Looks connected with #911.