Why interior mutability?

[mikebenfield]

I'm curious about the rationale behind allowing mutations in rlua types without mut? I get that it may be more convenient, but is there some justification as to why it's not just undermining the intent of Rust's type system?

[kyren]

First and foremost, let me apologize for the rant you are about to receive. Your question is a completely reasonable one, but one with a very complex answer, and I don't have time to write you a short, succinct answer.

Without interior mutability, you suddenly become unable to do a large number of things that it really would make sense for you to be able to do, and that Lua itself can do easily. That, and even if you do design an API that doesn't allow interior mutability, all it does is either end up being kind of a big lie, prevent you from holding onto multiple handles at once, or both.

Let's ignore memory safety for the time being (I'll get back to that). That means, we're pretending that any function call marked [m] in the Lua 5.3 reference manual never errors, and cannot mutate things, which simplifies things drastically.

If you make that assumption, there are a few things in Lua which are basically morally immutable, (speaking from the perspective of rlua here) raw_get, raw_len, pairs, sequence_values, thread status, creating Strings and Tables (I think you can actually violate this by having scripts mess with the globals table, I don't remember), creating userdata, getting userdata refs, and some others. So, it's definitely not nothing, but okay, say you have a Lua and you have a Table that immutably references the Lua because it contains a handle.

What can you do now? You can't call get, because that triggers metamethods and is definitely not morally immutable, you can't call set or raw_set, but okay that's expected, but you also start being unable to do really simple things, like convert values to/from Lua via ToLua / FromLua. Why? Well, ToLua and FromLua aren't restricted to only const Lua, so they can do things like create tables.. and then mutate them. In fact, sometimes that is the only sensible way to implement ToLua is to do gross procedural things like create a Table then fill it with values mutably. Or, even for FromLua, maybe you need to use get not raw_get because you want to write a type that converts from a table and accepts anything that acts like what you need, even if it is via metamethods?

Okay, you say, surely there's a way to have ToLua and FromLua without allowing mutation. Sure, for a lot of cases you absolutely could implement those without requiring mutation, so let's assume they're not a problem. Now, what can you do? Can you call a function with your table handle? You can't, because calling a function, resuming a thread, calling get/set, loading another script etc, are all mutable operations, you'd have to have a mutable handle to a Function to call it with your immutable Table, and then you'd have both a mutable and immutable borrow of Lua.

What ends up happening is, that you can only ever have two situations, either you can hold multiple references to Lua and not do much useful with them, or you can only hold one reference at a time and you're very limited. This limitation means you can't pass a handle-based value as a parameter to a function, you can't accept multiple handle-based arguments to a callback (without severe limitations), you can't indirectly trigger metamethods in conversions or in arguments to callbacks, it just ends up being a giant mess.

There's actually a project that explored this part of the design space pretty well, hlua. I'm not trying to disparage it, I mean I also don't let you do everything you might conceivably do with Lua via C either, but there are absolutely things that are very easy to do with rlua that are afaict impossible with hlua. If it suits your needs, and it is more to your taste, then I completely understand! Be warned though, and again I don't mean this as a value judgement agains hlua, but it appears to be missing a large amount of the safety focus of rlua, so it appears to be possible easily to cause UB with safe code using hlua, through at least 1) panicking from rust in a callback, 2) triggering lua_gettable (or other unguarded erroring functions) from within a callback which would jump over rust frames, or 3) any lua errors in a __gc metamethod or other sorts of memory errors. Lua is not an easy language to create a safe bindings system for.

OKAY, so what if you forego that, and loosen the restrictions a bit, and every type (mutable or not) actually just borrows the parent Lua immutably, and so solves most of the above problems. What you have now is a situation that just ends up being a big lie.

So, now you can have the following situation, say you have Table t and Function f, and maybe another Table tm, because you are unrestricted in the number of handles you can have. Only the f and tm are marked as mutable. That doesn't matter much, because secretly the Lua function represented as f closed over the table represented as t, and whenever you call it, secretly mutates t. In the same way, secretly table tm is actually a handle to the same table t, so if you mutate tm, you're just changing t behind your own back.

I haven't mentioned [m] methods yet! Since Lua 5.2, any function that triggers an allocation can in turn trigger a gc, and gcs can in turn trigger user defined __gc metamethods. This means that any call that even just allocates memory, like inserting into a Table, or even just creating a String, secretly may actually mutate any value you have a handle to.

Morally, you can think of Lua as a giant nest of Rc and RefCell, (but that cleans up Rc cycles).
There's not really a sane way, that I can think of at least, to represent that to rust using mutability xor aliasing, because everything is, in reality, both mutable and aliasing inside Lua itself. Where rust types are held inside Lua, they are held inside a RefCell and forced to simply make runtime assertions about their aliasing xor mutability.

tl;dr Lua looks at your attempts at sanity and control and immutability, and laughs in your face. The garbage collector can insert values into tables behind your back, everything can trigger mutation. This is fine, because that's just sort of what Lua is, so this API just accepts defeat and makes everything internally mutable.

Edit: I haven't actually tested the unsafety with hlua yet, because (unsurprisingly) I'm not an active user of it, so it's possible I'm wrong? If I am wrong, and somebody sees this and knows the bit I'm missing that makes it safe, please let me know.

Edit 2: I also want to clarify that the crate doesn't actually undermine rust's type system, because the crate has a goal (modulo bugs) of being completely 100% sound, which is actually no small feat with Lua. I can understand somebody having the viewpoint that it undermines the intent of the rust type system, however.

[mikebenfield]

Thanks for the extensive explanation. That makes sense; it sounds like a challenge to design around.

What do you think about this approach though: Consider each value from Lua, each Table and so on, to be just a handle referring to something inside a Lua value, rather than being the object itself. Then let each method that may mutate anything to do with a Lua value take a mutable reference to a Lua and if necessary an immutable Table or whatever. Things like get could continue to be methods on Table, and just take a Lua in addition to a Key, or they could be methods on Lua and take a Table and a Key.

Thus all your Tables are analogous to indices into a slice. It's the slice that needs to be mutable, not the indices. And the indices don't need to actually carry references to the slice.

Just a thought.

[kyren]

I think that such an API might match the mutability semantics somewhat more, but I'm not sure that makes it a better API. I guess you would have every possible method into rlua all be in the Lua struct? I'm not sure that's better.

I don't think rlua undermines rust's type system, because it is very up front about the interior mutability and is sound (modulo bugs of course), Lua definitely does not implement Send or Sync. Also, even if you have every method on Lua and take a &mut self of Lua, there is still the situation where callbacks have to manufacture a new Lua handle out of thin air. You now have a situation which is not actually unsound, but LOOKS really unsound where you will have multiple mutable Lua references that are all technically different instances of the Lua struct, but secretly all reference the same internal state. Therefore, you still need all the machinery inside Lua to handle interior mutability, you've just put an artificial limit on what the user can do with references to Lua, because the borrow checker will enforce aliasing restrictions it doesn't actually need to.

[jugglerchris]

Closing as this question has been answered.