RaptorJIT language side evolution and Lua compatibility

[tst2005]

Hello,

After the "Remove Lua C-API" discuss, If RaptorJIT choose his own road.

I wonder if it is easy to introduce in RaptorJIT :

• lexer/parser change to support new operator introduce in lua 5.3 (<<, >>, &, |, etc) and the implicite call of their meta.
• try to stay compatible at the lua code side

Regards,

[lukego]

Supporting bitwise operators like << >> & | would be great!

There has been some past discussion on the LuaJIT repo over at LuaJIT/LuaJIT#63.

Mike wrote that "the main implementation cost [is] modifying all of the vm_*.dasc files." This would not be a challenge anymore after we have ported the VM to C (#199) because then it would only be one C source file to update.

So perhaps we can revisit this issue once the VM port is done and then look at the best way to support these new operators (which may or may not have the same semantics as Lua 5.3 as per LuaJIT/LuaJIT#63.)

[tst2005]

Is it planned to have a partial control on the lexer/parser at runtime ?
For now RaptorJIT is like Lua/LuaJIT, working on a fixed lexer/parser (managed in the C source) modification can be only made at the compilation time.
We have something like macro? or a way to add new operator ?
It is just a question to jump outside of the basic need to run lua 5.3 code.
For sample, I dreams to add a custom != MACRO for the ~= operator ! ;-)

We can also transform foo << bar to bit.lshift(foo, bar) with MACRO ? it should be a way to produce change to support new feature with thirdparty (lua module) code.

I'd like to know your opinion about MACRO processing and non static grammar parser...

[lukego]

@tst2005 You are welcome to experiment with preprocessing and/or metaprogramming features on a branch and make that available for other people to merge :)

The nearest example that I can think of is DynASM. This is a preprocessor written in Lua coupled with a runtime library. Mike Pall did the original targeting C but @capr made a Lua version in LuaPower that lets you write inline assembler. The main limitation in LuaJIT/RaptorJIT seems to be not having a directive to preserve line numbers in the source file through preprocessing. It's a major pain when Lua errors use line numbers that don't match the source file.

This could justify a simple change to the RaptorJIT lexer to "hint" what the line number is in terms of the source file. On the other hand @CapsAdmin is working on a dynamic assembler that's like DynASM but without a preprocessor and I suspect this would make a better approach with no magic syntax. (The preprocessor is mostly there to make code generation fast but that's because DynASM was designed as a JIT backend rather than a macro-assembler.)