Using luau as a drop in for lua-5.1 and/or luajit

[JDaance]

I'm using an older game engine and in it I'm running lua-5.1 or luajit interchangably, and I made some tries to get luau working, but it seems there are quite a bit of things that I have to solve. I thought I'd ask for help here and document my problems in the event that the authors want to prioritize making this easy to do.

By just including luau in my cmake project I immediately run into these problems:

• Subprojects in pure C cant include lua.h since it uses default values for some functions like lua_pushcfunction, luau_load
• There is no lauxlib.h (it seems lualib.h is basicly the same though).

My strategy is to use whatever hacks I can to get issues like the above working, and to add some kind of shim project to re-introduce the io-module and other things that I need from standard lua, by basicly copying the lua-5.1 implementation. Any advice on how best to accomplish this would be very velcome! My main idea is to build luau using its headers, and try to use the standard lua-5.1 headers for dependencies, and just hope to solve all linker issues. Maybe i can trick luau to rename lua_pushcfunction with a macro, and not have to edit the source, or just rename it in the luau source and create a standard lua api wrapper.

I will try to do this and document my efforts here, as luau have some properties I have been looking for a lot, I especially welcome the low-overhead debugger with native file-line and step support, since the one I have currently written is too slow and I have not been able to get reliable stepping.

[zeux]

Subprojects in pure C cant include lua.h since it uses default values for some functions like lua_pushcfunction, luau_load

This should be fixed as of latest release; note that you'd need to define LUA_LIB as extern "C" conditionally though - we plan to include a separate config option for this in the later release.

[JDaance]

Wow that was quick! That indeed solved that issue, thanks!

I have lua-cjson, luasocket, luabitop and lpeg (all common lua libs) making a fuss about details now. I get lua_newuserdata, lua_pushcfunction, lua_pushcclosure needs an extra argument, I can probably work around this somehow, but its not drop in.

I get this compiler error, I have not tried to google it yet (maybe its some compiler flag?):

engine/3rdparty/luau-0.505/VM/include/lualib.h:7:33: error: void value not ignored as it ought to be
    7 | #define luaL_error(L, fmt, ...) luaL_errorL(L, fmt, ##__VA_ARGS__)

LUA_VERSION_NUM 501 is not defined so lpeg gives some error because of that

LuaBitOp wants #define LUA_NUMBER_DOUBLE in luaconf.h

luasocket wants this strange define from lauxlib.h: #define luaL_reg luaL_Reg

Creating a lauxlib.h like this seems to take care of the missing defines, I need to sort out the function arguments and error macro to get further now.

#pragma once

#define LUA_VERSION_NUM	501
#define LUA_NUMBER_DOUBLE
#define luaL_reg	luaL_Reg

#include "lualib.h"

[JDaance]

Got it to compile now by fixing many subtle differences and finally dropping lpeg and luasocket who used tons of weird intricacies. When I finally started it I ran into problems here where I load some external lua libs:

	lua_getglobal ( state, "package" );
	lua_getfield ( state, -1, "loaded" );
	luaopen_cjson ( state );  // for some reason cjson does not register a package
	lua_setfield ( state, -2, "cjson" );

It said something about cant index null for "loaded". It seems "package" is null? I can probably work around this, I just worry about how much luau has changed the usual lua require system 😅 I'll continue hacking away at this when I have time, and hopefully I can make some more comprehensive report when I get things running.

[zeux]

Yeah we don't currently include the package library at all, see https://luau-lang.org/compatibility#lua-51.

[JDaance]

Looking at this further it seems the entire lua require function is gone, is this documented anywhere? I will try to add the original function to my shim code and see it its possible

[JDaance]

Looking a bit closer I think lua-5.1 sets the env on the filehandle userdata somehow, maybe userdata inherits the creating functions env in standard lua? But env on the userdata struct seems to be removed in luau. Maybe I would be able to rewrite the io module to use different metatables for different filehandles instead of the standard env usage. Hopefully not many common lua modules uses this userdata->env technique

[JDaance]

Indeed lua-5.1 lua_newuserdata will grab current env and set it on the userdata, while luau has no env on userdata.

lua-5.1

LUA_API void *lua_newuserdata (lua_State *L, size_t size) {
  Udata *u;
  lua_lock(L);
  luaC_checkGC(L);
  u = luaS_newudata(L, size, getcurrenv(L));
  setuvalue(L, L->top, u);
  api_incr_top(L);
  lua_unlock(L);
  return u + 1;
}

luau

void* lua_newuserdatak(lua_State* L, size_t sz, int tag)
{
    api_check(L, unsigned(tag) < LUA_UTAG_LIMIT);
    luaC_checkGC(L);
    luaC_checkthreadsleep(L);
    Udata* u = luaS_newudata(L, sz, tag);
    setuvalue(L, L->top, u);
    api_incr_top(L);
    return u->data;
}

[JDaance]

I have now rewritten my lua-5.1 iolib copy to use different metatables for different filehandles, seems to work fine.

[zeux]

It's odd that iolib cares about the environment for userdata at all. I think you should be able to implement iolib with just a single metatable, maybe starting with 5.2 implementation would make sense. The only important deviation there is that iolib implements __gc to act as a last-chance cleanup for file handles, and Luau doesn't support __gc - this can be fixed by using lua_newuserdatadtor instead of lua_newuserdata though.

[JDaance]

I am not using a unique metatable for every handle, just 3 different depending on handle type. Looking at 5.2 might have been wiser though 😬 I did cheat and just commented out __gc, will switch to lua_newuserdatadtor when I get all major things running

[JDaance]

Skipping by the io.close for now, the next hurdle is a crash when the engine is trying to push a C++ function as a c-closure:

Illegal instruction seems to mean some kind of memory corruption, this is way over my head. It seems to run fine until this, but maybe the memory corruption occurs earlier somewhere, due to my hacks or how the engine uses lua.

[JDaance]

This is very promising! It indeed works to push the error handler and error message now, but instead it crashed on calling the pushed error handler like this:

I'm starting to guess "illegal instruction" is some kind of luau assert thing. Anyway I guess it is saying that this state->status is wrong and stops the function call? Do you have any ideas?

[JDaance]

This works, but it might not at some point, and I had to include lstate.h

#if ( WITH_LUAU )
    // luau will (rightly?) block function call in errored coroutine, we have to hack it
    this->mState->status = LUA_OK;
#endif
lua_call ( this->mState, 1, 0 );

If there is another way I would be happy to know!

[zeux]

Yeah I think by default our assertions are silent which we should change... they also can't be set from C code and require C++ code instead. If you have C++ code in your program then this will do it

#include "Luau/Common.h"

static int assertionHandler(const char* expr, const char* file, int line)
{
    printf("%s(%d): ASSERTION FAILED: %s\n", file, line, expr);
    return 1;
}

int main(int argc, char** argv)
{
    Luau::assertHandler() = assertionHandler;

[zeux]

As for calling functions, if your goal is to get the stack trace then using something like lua_debugtrace or an equivalent implemented using lua_getinfo would be the recommended path I think.

[JDaance]

Yes the safer way would be to get the stack trace by lua_debugtrace and then running the error handler on the main state. But this code runs now, I will see if I run into problems with it. My application is already frozen when a coroutine errors so I just need to fire off a crash report and not get a segfault in the process. If I get to a stable state with luau I can refactor some things to be more suited to luau and have other runtimes as the off-case.

[JDaance]

I actually got my game to run now, very promising!

Issues remaining for now:

• iolib not closing files (just skipped for now), will probably rewrite how the io module keeps track of close function
• had to run without luasocket and lpeg since they used obscure lauxlib stuff, will look into that later
• debug,getinfo is replaced by debug.info and this seems to give me nil for source and linenumber, probably some other fields, no big deal
• could not do the above push of a c++ function, no idea why. Since the game seems to run fine in light testing there is hopefully no general memory corruption and just something about that specific function push.

I am in general very happy with my shim project adding require, loadstring, loadfile, dofile and io module. I had to rename a few functions in luau to make it work, and conditionally define LUA_API as you said. I am looking forward to integrating my debugger after fixing the above 😎

[JDaance]

I have gotten luasocket to compile and load, not yet been able to verify that it works. Lpeg unfortunately relies on the same setfenv for userdata as iolib did, so I'm kind of stuck on that one. I tried running some pure lua replacement for lpeg to get past it and run my server but it uses every possible trick and function in the 5.1 api. I'm pushing the boundaries here on whats expected to do with luau, but I want to run my server unit tests in a good debugger.

[JDaance]

I might be able to rewrite lpeg to use a similar trick as I did to iolib, but this time with a new metatable for every userdata. Lpeg wants to use some newer lua_setuservalue to store a table for each userdata, but falls back to lua_setfenv for lua 5.1. It changes this table all the time, using it as some kind of storage mechanism. Maybe I could use the first metatable as this data value, and let the original metatable be a parent metatable. As far as I can tell this set/get uservalue should only be used as internal storage for lpeg and not affect behavior in lua code. Knowing absolutely nothing about lpeg is not helpful.

[JDaance]

I managed to get "lulpeg", a pure lua lpeg to run by implementing "load" and "loadstring" properly. Thats good enough for me, great lpeg performance is not important when I run my server tests, and the code will later run on luajit in openresty.

I can confirm luasocket seems to work so far, even though I know it uses __gc it seems to do fine for unit testing. Worst case I will hack it to use the destructor version.

[JDaance]

I have now completed my luau integration, debugger and all. It runs my game, passes all my tests, breaks on breakpoints and steps in code. But I have not and will not use it in a production capacity until I have been running it in development for a while, so I can't speak for crashes and things like that yet.

The only things that are not working and seemingly cant be fixed are:

• string.dump
• setfenv for userdata, used by lpeg and maybe others (cant use lpeg without rewriting things)
• usage of __gc in libs has to be rewritten to lua_newuserdatactor or care has to be taken to close files/sockets etc

In my shim project I have added:

• dofile
• load
• loadstring
• require
• module
• debug.getfenv
• debug.getinfo
• debug.getlocal
• debug.getregistry
• debug.getmetatable
• debug.getupvalue
• debug.setfenv
• debug.setlocal
• debug.setmetatable
• debug.setupvalue
• io.close
• io.flush
• io.input
• io.lines
• io.open
• io.output
• io.popen
• io.read
• io.tmpfile
• io.type
• io.write
• os.execute
• os.exit
• os.getenv
• lua_getallocf
• luaL_loadfile
• luaL_loadbuffer
• luaL_loadstring
• luaL_ref
• luaL_unref
• luaL_gsub
• aliased luau bit32 to bit

And a bunch of defines needed for compatibility. Most of these are the pure lua-5.1 implementation, but there are some rewrites here and there. Looking forward to debugging and profiling now 😎

[OlivierNicole]

Heh, SILE does use lpeg.

then patching for destruction and userdata values (setfenv on userdata) would be pretty hairy. At this point of complexity patching luau with setfenv support for userdata might be the best route

I'm not sure to understand this last sentence as this seems to be the only route to me (except giving up).

[JDaance]

Yeah well I learned what that sentence meant the hard way, running into problems with advanced third party libs :)

Lpeg and some other lua C-libs rely on using lua_setfenv on userdata with some arbitrary value, as some kind of bookkeeping mechanism, basicly using setfenv for something its not meant to be used for. In 5.2+ this is changed to lua_setuserdatavalue, which is much more clear. But neither version of values associated with userdata exists in luau.

I am not sure if lpeg uses __gc metamethod on its userdata, if so this might be (somewhat) easily patched to use luau's lua_newuserdatadtor and do the gc part in the destructor passed (this is how i did it in my io lib dropin).

Userdata values might be more complicated to fix, it should be possible to implement lua_setuserdatavalue outside of luau and patch/find compilerflags to make lpeg use that, as it does so for lua 5.2+. This implementation of userdata values would be less efficient and store the userdata value in a dict somewhere (table in the lua registry probably) but i doubt it would have measurable performance impact in the real world.

All this pushes the limits of my understanding of lua internals and may be more or less accurate 🙂

[OlivierNicole]

Maybe using lulpeg, like you did, would be easier then 😅

[mundusnine]

I have not yet uploaded anything, I would want to make a clean project out of my integration for that, but holler at me and I will send you a zip with what I have.

Hello, this is a holler. I am currently integrating a debugger for a project and would like to test out using luau and adapting LRDB for luau. I already integrated LRDB and invision using it with something like teal but would prefer having a faster more portable runtime so luau could be a better solution.

If ever you agree it would be really appreciated.

[ctemple]

I have not yet uploaded anything, I would want to make a clean project out of my integration for that, but holler at me and I will send you a zip with what I have.

Your works is great. I have a similar situation as you. Can you send me a zip file? I would greatly appreciate it!