Luau

[reflectrpteam]

This is an attempt to add a new Lua version along with the default one. In this PR you can find an implementation of an interface allows to switch between Lua 5.1 and Luau in runtime. Can be considered as stable but due to the complexity can potentially contain issues that are expected to be revealed after the release.

lua5.1 and luau libraries are shared now for all build configurations. But Lua_Server and Lua_Client are static. It makes the system flexible on the one hand and resolves the name conflict on the other.

You can enable Luau for a specific resource for client, server or both.

<meta>	
	<lua client="luau" server="luau" both="luau" />	
</meta>

Due to the flexibility of the interface new versions of Lua can be added in the future.

[CrosRoad95]

One thing from my side: if possible, download vendor files from luau repository to do not bloat mta codebase and make future updates easier,
example how you can do this you can find in mta bulletphysics pr :) it contains example for client and server side download of 3-rd party files
#1246

[reflectrpteam]

Unfortunately, we cannot use the default Luau code in the future. Some adjustments required in the code.

[CrosRoad95]

Unfortunately, we cannot use the default Luau code in the future. Some adjustments required in the code.

you can download luau and make a folder with patched files you copy onto luau files

This pr just show how much the mta team doesn't give a damn about this project

[Xenius97]

Instead of < lua > could be better to set lua version per script.
Just like < script src="server.lua" type="server" lang="luau"/ >
If lang is not set script uses 5.1 as default.

[Lpsd]

Instead of < lua > could be better to set lua version per script.
Just like < script src="server.lua" type="server" lang="luau"/ >
If lang is not set script uses 5.1 as default.

This goes against the current implementation whereby each resource has its own Lua VM (on client and server)

[MegadreamsBE]

Instead of < lua > could be better to set lua version per script. Just like < script src="server.lua" type="server" lang="luau"/ > If lang is not set script uses 5.1 as default.

Disagree, if you have a need to mix Lua versions you should split it up in multiple resources.

[MysterAnonimi]

Thanks

[Disinterpreter]

IMHO pros and cons:

Pros:

1. Good syntax and language
2. It is really popular and developed version or Roblox's lua. Maybe we can use their tools in future
3. Knowing developers made this (this user account hide people which we work many years)

Cons:

1. It won't work for our luac.mtasa (but you can cache scripts)
2. Diffrent VMs, like you can't write on Lua and Lualu in one resource (but i think exports for multiple resources are working)

My opinion - lgtm.

[dmi7ry]

Not finished, but already usable
https://github.com/dmi7ry/Luau-definition-for-MTA

[Pirulax]

Unfortunately, we cannot use the default Luau code in the future. Some adjustments required in the code.

you can download luau and make a folder with patched files you copy onto luau files

This.
Making manual patches all the time is not a great idea.
And I'm sure keeping Luau up-to-date is preferred.
Also, what are the changes made? Are they really necessary?
Maybe we can do some workaround to avoid having to modify it.

I want to admit that we don't really have any "test branch" which mta should have. we could use huge playerbase to test >?
features, make in main menu so kind of information to switch to "testing" version of mta if you want to help mta bla bla bla.

I guess there are nightlies, but even for that the PR has to be tested (and be somewhat stable).

---

So anyways, let's get the ball rolling.

I'd like this to be merged, unlike my attempt at LuaJIT (Also, Luau is better maintained, so it's superior anyways).
My idea for testing is as simple as taking some opensource mta mod, running it, and testing if it works.
If it does, then we can make a nightly out of it, and encourage a few servers to start using it (I'm pretty sure there'd be people willing to help, there always are, shoutout to them!).

On another note...

Before we do all of that, we need some netcode stuff too for checking if luau scripts are signed, until then compiled scripts are a no-go, executing untrusted bytecode is dangerous (Same reason why currently Lua scripts go thru MTA's site, they get signed too)

[tederis]

But I don't seen anything like it in Luau's code tho.

This is the reason for the excessive memory usage of Luau. Luau doesn't has LUAI_EXTRASPACE feature so I was forced to implement it on my own by allocating an extra 4 bytes for every memory page. And this must be removed in future by commiting to the Luau project directly.

[Pirulax]

Ah, okay so I didn't know about LUAI_EXTRASPACE. I can use this to store the CResource* of the state, instead of editing the Lua source code. Thanks.

In general, Luau requires more adjustment because currently it uses more memory than it could
You mean it uses more than it should?

Could you explain the last 2 changes? (luai_userstateopen luai_userstatethread) It doesn't make a whole lot of sense to me.

Yes, it uses more than it should, right.

luai_userstateopen is called at the end of lua_newstate so you can write a custom data just after the lua_State creation. In the context of this PR this macros adds a "magic" number 0x5989C266 before a lua_State object(that's why LUAI_EXTRASPACE was used). 0x5989C266 is just a number for the unambiguous lua 5.1 recognition. (Luau at its own side is also pushes some "magic" data before a Lua state.).

luai_userstatethread is called when a new coroutine is created. In the context of this PR this macros copies a "magic" number to a new coroutine Lua state.

That is the answer to my problem, thanks.
Luau doesn't have LUAI_EXTRASPACE, but it uses the userdata that can be given to every state in lua_open.
So that's what I'm going to do for both Lua 5.1 and Luau.

Also, is this 0x5989C266 value official, or we came up with it?
Would be nice to have it in a macro, like LUA_51_MAGIC.
I assume Luau has a macro for it's magic number.

[tederis]

Ah, okay so I didn't know about LUAI_EXTRASPACE. I can use this to store the CResource* of the state, instead of editing the Lua source code. Thanks.

In general, Luau requires more adjustment because currently it uses more memory than it could
You mean it uses more than it should?

Could you explain the last 2 changes? (luai_userstateopen luai_userstatethread) It doesn't make a whole lot of sense to me.

Yes, it uses more than it should, right.
luai_userstateopen is called at the end of lua_newstate so you can write a custom data just after the lua_State creation. In the context of this PR this macros adds a "magic" number 0x5989C266 before a lua_State object(that's why LUAI_EXTRASPACE was used). 0x5989C266 is just a number for the unambiguous lua 5.1 recognition. (Luau at its own side is also pushes some "magic" data before a Lua state.).
luai_userstatethread is called when a new coroutine is created. In the context of this PR this macros copies a "magic" number to a new coroutine Lua state.

That is the answer to my problem, thanks. Luau doesn't have LUAI_EXTRASPACE, but it uses the userdata that can be given to every state in lua_open. So that's what I'm going to do for both Lua 5.1 and Luau.

Also, is this 0x5989C266 value official, or we came up with it? Would be nice to have it in a macro, like LUA_51_MAGIC. I assume Luau has a macro for it's magic number.

No, 0x5989C266 is the completely custom invention. It just a way to differentiate Luau and Lua 5.1 by having just a pointer to a Lua state.

[Pirulax]

Okay, so we ended up deciding that adding an extra size_t to the beginning of lua_State would be a good solution.
The same will be done for Luau too (since we have a submodule for it), it's a really small change (3 lines in total).

[Pirulax]

Update: Adding anything before the header probably won't work. I tried adding LUA_EXTRASPACE to Luau, but that isn't possible (easily) right now either (As, unlike in Lua, in Luau the lua_State objects are gc objects, not raw mem allocs, and luaM_freegco expects a pointer directly to a gco, not an altered raw pointer.
I guess another option would be to add store this info in the beginning of CommonHeader somehow. (Nvm, waste of memory)

[Pirulax]

Okay, I give up.
I think we have to make a compromise here.

1. We keep Lua 5.1 at a cost of a higher memory (extra 4-8 bytes for every Lua allocation) and CPU (due to the translation layer - vlua) usage.
2. Drop Lua 5.1 support, have only Luau - Luau supports Lua 5.1 scripts, BUT not bytecode - Ergo, all compiled scripts would cease to work. This was brought up many times by @sbx320 and me (But for LuaJIT), and I always vouched for people simply re-compiling their scripts - As anyone who obtained those scripts in a "legal" manner, should be able to recompile them without any issue. (Expect that there are a few community scripts that are distro'd like that).
3. Similar to 2. but build a transpiler from Lua 5.1 VM bytecode Luau - Let's be realistic this is way too much effort, nobody's going to do it
4. No Luau - Shit option, I advise against it - Luau is the future, it's just an upgrade in both performance and productiveness.

I'd honestly go with 2. Main reason is performance and a memory usage - Merging something that could drastically increase memory usage could harm servers that are already on the edge of that 4GB limit.
I haven't tested, but with this change I'd anticipate at least a 20-30% increase in Lua VM memory usage. This would also affect cache lines heavily (I really hope TValue's aren't padded with this, if they're 2. isn't even an option imo, as the performance impact is too heavy).

I think higher performance+less memory usage would be totally acceptable for the cost of having to recompile your scripts.
As for servers that don't have the original sources, they can always stick to an older version that has Lua 5.1, not Luau.

[MegadreamsBE]

From my standpoint option 2 is the best. As you said, if the scripts were obtained legally it's easy enough to re-compile them, not a big deal. The pros outweigh the cons in this scenario.

[szkiddaj]

Same, 2. option is the best. Just the new features (continue statement, compound assignments, etc..) alone could help a lot writing better codes.

This is a no brainer in my opinion, if it’s possible.

[sbx320]

We keep Lua 5.1 at a cost of a higher memory (extra 4-8 bytes for every Lua allocation) and CPU (due to the translation layer - vlua) usage.

Why do we need a translation layer if we ship both Luau and Lua 5.1 and use the respective VM depending on the resource setting? In both cases only one VM would be running and Lua(u)-objects would never hit another VM.

2 is not a realistic option. There are too many compiled scripts out there that'd just break, where source isn't available because no one ever cared about it. At that point we'd just be creating another MTA fork.

Merging something that could drastically increase memory usage could harm servers that are already on the edge of that 4GB limit.

Those should move to x64 anyway.

[MegadreamsBE]

2 is not a realistic option. There are too many compiled scripts out there that'd just break, where source isn't available because no one ever cared about it. At that point we'd just be creating another MTA fork.

A line has to be drawn somewhere for the sake of progression. We just can't keep hindering ourselves to maintain backwards compatibility for outdated scripts that are more likely to be insecure (and with the source lost you can't even verify it) anyways.

We need to force the community to improve and ensure MTA does so right alongside them.

[sbx320]

Isn't it fair to say that if someone didn't care about source code of the resource that they use, they wouldn't even probably try to update their MTA server binary?

Not really. Many people update their servers that have no idea how to do anything in Lua. Also this would break client scripts as well, for example in various race maps.

A line has to be drawn somewhere for the sake of progression.

Yes. However the benefit of breaking compatibility also needs to be weighted against the downsides of it. I don't consider Luau to be that significant of an upgrade over standard Lua for it to be worht breaking all existing compiled scripts.

• Option 1 seems managable in terms of development effort and future maintenance, although I'm still not sure why there would be any relevant runtime overhead if we select the VM on resource load.
• Option 2 isn't happening.
• Option 3 is unlikely to work (even if we find someone to build a translator) given that Luau doesn't have a compiled format (and going 5.1 bytecode -> Luau source is likely to be very slow).
• Option 4 seems most likely.

[ffsPLASMA]

Regarding compiled scripts (from race maps or whatever), if mta team would provide a fast batch decompiler of luac to get the readable pseudocode back and potentially recompile it for luau, then this could work.

[Pirulax]

Those should move to x64 anyway.

GTA is still 32bit, and will be until we finish reversing it. So x64 won't help.

Why do we need a translation layer if we ship both Luau and Lua 5.1 and use the respective VM depending on the resource setting? In both cases only one VM would be running and Lua(u)-objects would never hit another VM.

Because we need to know which lua_State is which Lua version.
Obviously the 2 lua_State's aren't compatible, thus we have to virtualize the calls (and that's what this PR is doing), but that inherently comes with a performance loss.

New: Option 5

We would increment the version to 1.7, this way everybody gets to keep MTA with Lua 5.1. Server owners with compiled scripts without sources would be stuck on the last 1.6 version, and everybody else would be on 1.7 enjoying Luau (And let's be real, majority of players play on servers that will be able to move to 1.7). Also, server owners not having sources and not know much about Lua wouldn't have much use anyways of newer versions - They can't take advantage of new features, if they don't know how to script in Lua, no?

[tederis]

Why do we need a translation layer if we ship both Luau and Lua 5.1 and use the respective VM depending on the resource setting? In both cases only one VM would be running and Lua(u)-objects would never hit another VM.

VM selection on resource start is the best(from the perfromance perspective) solution. But how does this can be implemented? if-else statements for every lua function call site? The strategy pattern? It will require a lot of refactoring. And that is the reason why the translation layer system was chosen. It requires almost no modification of the existing MTA codebase, works as fast as possible and can be easily extended.

[AlexTMjugador]

But how does this can be implemented? if-else statements for every lua function call site? The strategy pattern? It will require a lot of refactoring. And that is the reason why the translation layer system was chosen. It requires almost no modification of the existing MTA codebase, works as fast as possible and can be easily extended.

I didn't review this PR deeply enough to know if I'm about to say something impractical, but why can't we just default to Lua 5.1 for resources that don't explicitly opt-in to Luau in the meta.xml file?

If we ship both Lua VMs, and only create the appropriate one after parsing the meta.xml file, that sounds reasonable enough. If communication between VMs of different versions is necessary, we could look into using userdata instead, perhaps?

[tederis]

But how does this can be implemented? if-else statements for every lua function call site? The strategy pattern? It will require a lot of refactoring. And that is the reason why the translation layer system was chosen. It requires almost no modification of the existing MTA codebase, works as fast as possible and can be easily extended.

I didn't review this PR deeply enough to know if I'm about to say something impractical, but why can't we just default to Lua 5.1 for resources that don't explicitly opt-in to Luau in the meta.xml file?

If we ship both Lua VMs, and only create the appropriate one after parsing the meta.xml file, that sounds reasonable enough. If communication between VMs of different versions is necessary, we could look into using userdata instead, perhaps?

In this PR the only one VM exists at the same time for each resource. And Lua 5.1 is already default. Without the virtualization we simply won't be able so dispatch the Lua function calls.

[AlexTMjugador]

In this PR the only one VM exists at the same time for each resource. And Lua 5.1 is already default. Without the virtualization we simply won't be able so dispatch the Lua function calls.

Right, I understand this overhead is needed because we want to instance the same, multi-version VM across resources. But what is preventing us from using different VMs for different Lua versions for different resources?

[tederis]

It seems that this PR and its implementation details are unclear for most of the people. It adds a custom translation layer (called vlua) that replaces Lua API functions by its own "proxy" versions. So, lets say MTA is calling lua_pushstring, which is actually a vlua function that internally deduces the version of Lua from a lua_State pointer. And then using that information it invokes the corresponding Lua function which can be Lua 5.1 or Luau. Currently the Lua version is deduced by a special "magic" field that's placed before the lua_State object. Simple, fast, scalable.

[AlexTMjugador]

Thanks @tederis, that explanation made it much clearer to me! I think it's a fundamentally smart approach.

[sbx320]

But how does this can be implemented? if-else statements for every lua function call site?

Some if-else and a bunch of templates. It's a bit more work, but it doesn't sound too bad.

I would've reimplemented the Lua API as template<LuaVariant>. So that lua_tonumber<Lua51> calls lua_tonumber from 5.1. Most API usages could then also be templated (e.g. CLuaArguments) to catch existing usages. Legacy Script functions would also become templates (new functions via argparser could support this directly), instantiated once for each VM type.

That way we can avoid any runtime CPU overhead and only gain some code size.

[TheNormalnij]

We would increment the version to 1.7, this way everybody gets to keep MTA with Lua 5.1. Server owners with compiled scripts without sources would be stuck on the last 1.6 version, and everybody else would be on 1.7 enjoying Luau (And let's be real, majority of players play on servers that will be able to move to 1.7). Also, server owners not having sources and not know much about Lua wouldn't have much use anyways of newer versions - They can't take advantage of new features, if they don't know how to script in Lua, no?

They can install resources with new features without Lua knowledge.

It's allowed to use EmmyLua with MTA to have all benefits of typed language. Or you can use TypeScript -> Lua transaction layer and have significantly better language that Luau. This options request no changes in MTA codebase.

Lua community provides a lot of native modules for Lua. LuaRock is partially integrated in MTA and provide some multi-threading support for windows.
Also debugging tools can be used with vanilla Lua.

There is also a lot of ways to use Lua knowledge. Defold uses Lua as main language. Project Zomboid, Garrys Mod, BG3 SE use Lua for mods

I am not a fan of this changes. EmmyLua + debugger provide the best Lua developing experience for me. Maybe somebody is going to add Luau in MTA. Just please, don't break old resources. Luau has not enough benefits for it.

[Pirulax]

It's just the way MTA works.
We deprecate functions, then keep them for 10 years at which point it's impossible to remove them due to "backwards compatibility".
When there's a new feature nobody wants to merge it, because "it might break old scripts" (Bonus is that those scripts are compiled so they can't even be fixed).
This endless backwards compatibility drags the whole project back. We aren't Microsoft.
And sadly it also fragmentates the community, because so many changes just aren't ever added to this repo, that people just end up making forks where they can freely merge the PRs from this repo (Bullet Physics for example).

It's allowed to use EmmyLua with MTA to have all benefits of typed language. Or you can use TypeScript -> Lua transaction layer and have significantly better language that Luau. This options request no changes in MTA codebase.

Lua is slow. Luau is faster. LuaJIT is the fastest, but security is questionable (The main reason why Luau isn't JIT).
Luau is a drop-in replacement with TYPING, and lots of other features that improve dev. experience out-of-the-box, not to mention that it's faster than Lua in a lot of scenarios, and doesn't force you to write unreadable code just to be faster (numeric for loop vs ipairs, etc).

[AlexTMjugador]

For what it's worth, I think the approach outlined by @sbx320 seems worth exploring to move this PR closer to being merged, in addition to moving Luau to a submodule whose changes can be easily tracked under version control.

I find that @TheNormalnij has some valid points about Luau's addition not being strictly necessary to MTA, but given that:

• Luau is well-maintained and battle-tested by Roblox, one of the most popular MMOs in the world, for a very similar use case,
• We've discussed workable ideas to address backwards compatibility and runtime overhead concerns,
• It's a lightweight library (a few MiB big judging from the size of the official luau-analyze.exe build, which is comparable to the average website size),
• Its addition is mostly a one-time effort (and it could pave the way to implementing ways of reducing the required manpower to maintain vendor components; see below),
• It would serve as an enabler for things everyone wants: more complex client-side scripts (c.f. @MegadreamsBE's Sphene and console emulators, which are unlikely to have performance good enough to be used in production without the aid of a fundamentally faster runtime), and proper vendor dependency management in MTA (Luau can be used as an example of how to get things done when it comes to adding vendor code),

I think realistically there is no downside to this PR, so I would recommend against letting this good effort go to waste. I would indeed advocate for caution in adding big features like Bulletphysics, because those are either complex in themselves or open cans of worms in terms of future complexity, practical usability in their MVP form, expansion, and maintainability, but this kind of "infrastructure" improvements pack, in my opinion, a lot of bang for the buck that's worth the relatively little effort.

[Pirulax]

Also, something LuaJIT, and Lua 5.1 doesn't have is native vectors, that cause no GC pressure.
Vectors are something you really don't want to use currently, because of how much they tank the performance (because of the gc pressure).
Luau was literally designed for MTA :D
If it's good enough for Roblox, it's good enough for us.

[TheNormalnij]

We deprecate functions, then keep them for 10 years at which point it's impossible to remove them due to "backwards compatibility".

Sometimes we do very small changes in API. This changes doesn't breaks much and old resources can be adopted. This is ok for me. Lua is a very sensitive part of MTA. And changes in bytecode break a lot of old resources. There is a lot of script markets, where server owners buy compiled scripts. Some freelancers sell compiled resources only. I think we are going to break too many things.
This is strong enough reason to keep Lua 5.1 for me.

Lua is slow. Luau is faster. LuaJIT is the fastest, but security is questionable (The main reason why Luau isn't JIT).

In my experience Lua is fast enough. MTA performance is suffering from bad decisions in API + LuaVM <-> engine serialization.
It's nice to see real benchmarks. Significant performance impact is more like reason to add Luau as second LuaVM.

Bad API

• People uses raw and slow dx functions instead of any library.
• .. or ever CEF for GUI
• Objects in MTA are slow. They use dynamic objects pool and additional MTA streamer over it,
• All DB functions sucks. dbQuery provide some sort of queue for queries. This decreases maximum performance.
• EngineReplaceModel doesn't work well.
• fileRead is blocking operation. It copies file content between MTA engine and LuaVM. With encryption function and engineReplace*** function this is huge loss.
• toJSON is joke
• Vectors are implemented in wrong way

Also, something LuaJIT, and Lua 5.1 doesn't have is native vectors, that cause no GC pressure.

#1728 we can change our Lua implementation and do event better for MTA.

[Fernando-A-Rocha]

Keep Lua as it is, add Luau (optional) that can be enabled in meta.xml
Everybody is happy, end of story ❗

[Rilot06]

Keep Lua as it is, add Luau (optional) that can be enabled in meta.xml Everybody is happy, end of story ❗

Others have said before that if we keep 5.1, it would need a translation layer, causing more cpu usage and it would use more memory, I don't think it's an option, performance isn't that good currently either, and it would be even worse if we keep 5.1. If we keep worrying about backwards compatibility, the game will never get a decent upgrade, I don't see why the game has to support 15 years old compiled (mostly stolen) scripts. 99.9% of the time if you are a server owner/developer, you wouldn't backup just the compiled scripts, you would backup the sources too, only if you didn't steal it from other servers of course..

[TheNormalnij]

I think we are talking about two parallel version without big performance impact. Transaction layer is too hard

[Rilot06]

I think we are talking about two parallel version without big performance impact. Transaction layer is too hard

I don't understand what you are saying, there would be a pretty big performance hit if we keep both languages, Pirulax said it already.

1. We keep Lua 5.1 at a cost of a higher memory (extra 4-8 bytes for every Lua allocation) and CPU (due to the translation layer - vlua) usage.

Also the translation layer is literally the chosen method in this pr, it's already working, so I don't get what you are trying to say.

[TheNormalnij]

Ah, i forgot this information, sorry