A Lua VM in Rust

[naturallymitchell]

Coming from #21 (comment), here are a few projects I found that could help with a native implementation:

Lexer / Parser

• nom-lua53
• nom-lua
• syn (complete programming language example)

Interpeter:

• monkey-rust
• miri (not sure if this has any relevance to scripting)

Other non-PuC Lua projects:

• go-lua
• ravi
• shine

[jonas-schievink]

This is hopefully not in the scope of this project, bindings are hard enough, writing a correct implementation is a massive undertaking.

Apart from the fact that Lua's grammar isn't in LR(k) (so we can't use a nice parser generator like lalrpop), the most complicated things are probably the VM and GC, none of which are really reusable. The VM has to support metamethods, errors and stackful coroutines, while the GC needs support for weak tables (which can have 3 different modes) and finalization.

Whatever you'd end up with would likely be much slower than the reference implementation anyway.

Personally, I'm only interested in work in this area to be able to compile Lua to JS for a pet project.

Oh, and a long time ago I've worked on this project to learn Rust, which implements a Lua dialect. Can probably be used as a starting point, but it's extremely broken and unsafe, so maybe it's better to start from scratch.

[naturallymitchell]

Oh.. Thanks for breaking it down. What an undertaking that would be. That's interesting you had some work on this, but now I can see how bindings make the most sense.

[kyren]

I definitely think jonas-schievink knows more about this than me, but I just wanted to say I'm also sorely tempted to make my own Lua vm in rust. He's right that bindings really probably do make the most sense and a reimplementation would likely be slower than the reference vm, but the tempting part is that in order to make safe bindings, there is SO much overhead :/. I think it would be interesting to compare a rusty implementation of Lua that allowed for more lightweight bindings to rust vs stock Lua and rlua, and I think when bindings became a bottleneck that it would NOT be very hard to beat rlua + Lua at all.

I could make rlua faster of course, but there are just fundamental limitations here that I don't have any good plan of how to solve. The overhead of the registry is the big one, and it's really tempting to think that it should be easy to make a lower level safe API that uses the stack or at least prevents values from being moved to/from the registry, but it's just REALLY hard to come up with a consistent safe API that exposes the stack that doesn't impose huge limits on what you can do. hlua is where you end up if you go that route, and the problems there are the reason for the existence of this library in the first place.

I'm not trying to criticize vanilla Lua at all, please don't take this the wrong way when I say it, but Lua has one of the MOST C apis that I've ever used. Making it safe, type safe and panic safe and leak safe etc, is SO hard, that I guess what I'm saying is I understand the desire to just write an interpreter for Lua from scratch.

I'm definitely still going to work on rlua for the foreseeable future, but if I can find time I may also take a stab at a rust Lua VM, but it would probably not be something that would replace rlua anywhere in the near future at least, if ever.

[thelearnerofcode]

It could be possible to slowly chip away at lua and replace some parts with rust.

[naturallymitchell]

compile Lua

Does this new generic purpose Just-In-time compiler for Rust bring anything to the table?

[jonas-schievink]

@mitchtbaum Sure, that could be used to implement the VM. Might be interesting to see if some day a Lua implementation (or any dynamic language implementation) will outperform LuaJIT, but that's a long way off.

[naturallymitchell]

A couple more things to watch:

the most complicated things are probably the VM and GC, none of which are really reusable

Based on HN's HolyJIT discussion, it will use Cretonne, and in a Verifier issue there, a discussion about a Python JIT also includes GC plans. This low level stuff goes over my head, so I can only say rust-gc might also relate since it has a Javascript mention.

Lua's grammar isn't in LR(k) (so we can't use a nice parser generator like lalrpop)

lua-parser looks like it can build an AST and has strong roots. Then perhaps it could run on this stack in later steps.

[UserAB1236872]

I think at the point where you're doing what @kyren is talken about -- changing the entire Lua implementation to be Rust/rlua friendly, safe, etc you've hit a point where it's probably more prudent to just use a project with a Rust-focused embedded scripting language like Dyon or whatever. I'm using rlua for two primary reasons:

1. Lua is more mature, tested, tutorialized and so on than any new scripting language
2. Since I'm releasing my project to researchers who just want to do research, Lua is a lower barrier or entry due to the prevalence of information (sort of an extension of 1).

However, doing a ground-up reimplementation of Lua to just to play nicely with Rust bindings loses a good chunk of the perks from 1. Of course, there's still the language grammar, documentation, and tutorials, but it will by nature be less tested, and diverge in behavior in strange unknown edge cases. Especially if it's only used in this project, that means there are fewer people testing and interested in maintaining such a project.

Of course, there's always the option of "feature gates". Use a feature gate to control using the current C Lua or the experimental Rust Lua, but that still means needing to walk on eggshells in #[cfg(c_lua)] meaning you don't really gain much in terms of making usage nicer, you've just added two targets to maintain.

It's really not just speed that's the issue here, so much as Lua is a game industry standard and thus extremely well battle tested in a way that a ground-up Rust implementation just can't be, and wouldn't be for a very long time, especially when there are already more Rust-friendly embedded languages competing for usage attention in the Rust space. I think living with vanilla Lua is probably (and unfortunately) the best option.

[jugglerchris]

I'm closing this issue as I think it's out of scope for rlua, and has effectively been spawned off as https://github.com/kyren/luster .