Loading code in luau breaks in strange ways when pausing the main thread using the rfd crate

[setzer22]

Hi! I've found this issue in my project, blackack. It had been bugging me for a while, but I wasn't able to isolate a reproduction until today, where I finally had an idea of where the issue might lie in.

I noticed that on my application, code hot reloading broke after loading or saving a new file. I'm using the rfd crate to open a "pick file" dialog, and since I'm lazy and hadn't bothered setting up anything async just yet, I'm doing that synchronously on the main thread.

Now, what I've found out is that pausing the main thread by opening a pick file dialog with rfd leads to some sort of bad interaction inside mlua that... breaks things? After calling an rfd function, any subsequent calls to load will fail with incorrect parse errors. I can't be sure if the error is in mlua or rfd but this kind of feels like some UB is happening somewhere? 🤔

I was able to isolate this to the following minimal example I'm loading some code, calling rfd, then loading the same code again. If you run the example on Linux (simply dismissing the "open file" operation works), you will see how the second call to load panics due to the unwrap, with the following error:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: SyntaxError { message: "[string \"/home/user/cargo/registry/src/github.com-1ecc6299db9ec823/mlua-0.8.1/src/lua.rs:1352:32\"]:1: Expected identifier when parsing expression, got '3.1415926535'", incomplete_input: false }', src/main.rs:6:52

I'm not sure where the error lies in, but is the lua interpreter somehow runnning on a separate thread? I was thinking that perhaps the kind of "pausing" of the main thread done by RFD causes an issue with that, but honestly I'm at my wit's end on this one.

[setzer22]

Oh, and I just found out something else: This issue is only reproducible in Luau. I tried both lua52, lua53, lua54 and luajit and my reproduction example works fine in those.

Perhaps there's some extra considerations that need to be taken with respect to Luau and thread safety?

[setzer22]

Even more fun! 🎉 It seems I can only reproduce this on the gtk-3 backend of RFD, so I'll be also reporting the issue there. Maybe something's wrong with their usage of the gtk-3 bindings?

[khvzak]

Sorry, cannot reproduce on macos and ubuntu 22.04 (jammy)

[setzer22]

Sorry, cannot reproduce on macos and ubuntu 22.04 (jammy)

Ok, that's something then 🤔 Not being reproducible on Ubuntu supports the theory that it might be caused by the gtk-3 in rfd, but we still don't have any strong evidence on that. It's just the only dependency that would vary across machines that I'm aware of.

Either way, I understand this might be a very hard issue to debug, so unless you have any ideas about potential cases of UB in the library, or Luau iself (potentially, when interacting with GTK-3's event loop?), I'm not sure there's much else we can do at this point 🤷

For now, looks like my issue is solved after moving to the xdg-portal backend of rfd.

[khvzak]

What is your linux version? I can try another dist.

Perhaps there's some extra considerations that need to be taken with respect to Luau and thread safety?

I don't think there is anything related to thead safety. The code is run inside a main thread. Difference between Luau and other versions is a dedicated compiler. And looks like the memory after opening dialog is corrupted somehow.