Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Usage of TypeId to store type "references". #126

Closed
SirHall opened this issue Jan 29, 2022 · 4 comments
Closed

Usage of TypeId to store type "references". #126

SirHall opened this issue Jan 29, 2022 · 4 comments

Comments

@SirHall
Copy link

SirHall commented Jan 29, 2022

TypeID id's are not stable, as it is still not reliably collision resistant, and the hash generated is expected to change between Rust versions: rust#77125.
This makes it a poor candidate for serialization of user type references that is expected to survive longer-term storage.
It may be better if we want a longer-term hardened solution may be to utilize the typetag or inventory crates to allow simple serializable type references.

@SirHall
Copy link
Author

SirHall commented Jan 29, 2022

This could be extended to also store references to rust functions.

@khvzak
Copy link
Member

khvzak commented Jan 29, 2022

Could you please explain more? What did you mean under "user type references" and serialization of them?

There are few places where TypeId is used.

  1. To attach arbitrary data to a Lua instance via set_app_data. And this is a runtime feature and does not matter that hash can change between Rust versions.

  2. To store UserData. In this case I would expect to have a uniq hash even per type / per application or module. Having hash changes between rustc version is rather advantage than something wrong. Also it's an important feature when loading Rust/mlua modules to Rust/mlua application. Even same datatypes (UserData) should be independent and not shared between application and a module. Rust does not provide stable ABI and it would be highly unsafe to access them across ffi boundary.
    Although there is a nice crate abi_stage that provides such functionality.

If talk about UserData serialization it's fully suported using serde feature and does not depend on TypeId hash. Or did you mean something else?

@SirHall
Copy link
Author

SirHall commented Jan 29, 2022

Forgive me, I probably should have explained in more detail.
Lua Eris, is a modified version of Lua 5.3.5 which gives the developer the ability to serialize the state of the Lua VM itself to a byte stream and subsequently, write it to a file.
You can later then read in this byte array, push it onto the Lua stack and have the Lua instance resume perfectly from its previous state.

However, how would Lua serialize pointers to external objects?
Serializing pointers to functions simply won't do.
This is a C API, so the pointer doesn't store any type data with it for (light) userdata.

What you must provide also provide when serializing is a 'perms' table onto the stack.
The 'perms' table maps pointers generally into a string 'name', and so any serialized reference to an external data is named with this string.
Then when deserializing, you pass in an inverted table, mapping those names back into pointers.

Now given the various layers of safety abstraction, it may get a little more complicated to ensure that a developer using this writes their code in such a way that they can easily re-create external data, and remap to external functions when deserializing their Lua instance - so it may be better to implement this in a lower-level library.

So yes, the issue with TypeID is that if a developer were to distribute a new release, it is very possible that it will break the end user's serialized Lua instances.

@lenscas
Copy link

lenscas commented Dec 23, 2022

So yes, the issue with TypeID is that if a developer were to distribute a new release, it is very possible that it will break the end user's serialized Lua instances.

But even if something else than a TypeID was used, if you have a state that uses version X for a library and then use this state in an environment where version Y of said library is used can produce very big problems if the types changed. It wouldn't even surprise me that the fact that Rust has no stable ABI on its own can already mess things up for you in a big way.

And even if the types stay compatible in layout, the exact meaning of each field may have changed drastically causing invalid states anyway.

if such a feature is something that you would like to see supported then I think it would require much more than just a change in how mlua tracks which types are what, if nothing else it requires some change to make developers aware of this feature existing and writing types in such a way that they can handle being fed old versions of their types.

@khvzak khvzak closed this as completed Jul 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants