Runtime error in lrpar::ctbuilder::_reconstitute with target wasm32-unknown-unknown

[pablosichert]

Hi there!

I'm trying to parse a grammar from within WebAssembly, but

grmtools/lrpar/src/lib/ctbuilder.rs

Line 835 in f16a9d0

let stable = deserialize(stable_buf).unwrap();

produces the following error at runtime:

invalid value: integer `17471392538647552630`, expected usize

Looks like in some part of the library the target architecture wasn't taken into account correctly.

Do you have a guess where the error could be?

Please let me know if I there's some more information that I can provide to debug this issue.

[pablosichert]

You can find a minimum reproducible example here: https://github.com/pablosichert/grmtools/tree/wasm-example/examples/wasm.

[ltratt]

I've quickly tried to build your example, but I lack several of the tools to do so I'm afraid and, even if I did, there's a lot of stuff that's unfamiliar to me.

That said, my guess is that you might be trying/using wasm in 32-bit mode (17471392538647552630 is too big a 32-bit int). grmtools is untested on 32-bit (I haven't seen such a machine in years) so that is a potential issue. However, if I had to guess, I think it's more likely that part of your tool chain is using 64 bit stuff and another part 32 bit stuff (perhaps you're generating 64-bit stuff and running 32-bit stuff?).

[ltratt]

Aha, if I'd noticed the title of this issue, I'd have saved myself some work :) "wasm32-unknown-unknown" (i.e. 32 bit wasm), so that's definitely part of the problem. As to whether grmtools/serde is causing the problem or whether it's some part of your tool-chain I have no idea! But I would first try running wasm in 64-bit mode and see if that makes the problem go away.

[pablosichert]

but I lack several of the tools to do so I'm afraid and, even if I did, there's a lot of stuff that's unfamiliar to me.

I expected this to happen - unfortunately there really is a huge chain of tools and dependencies needed to get it working.

I don't want to waste your time digging into all the tooling, I'll investigate a bit first and come back with some more specific questions.

(In case you are still curious, I'll pack it up all together into one Docker image to ease the installation.)

But I would first try running wasm in 64-bit mode

That was my first reaction as well - unfortunately WASM doesn't support 64 bit yet, and likely won't for the nearer future — note that 32 bit strictly refers to the address space only, we still have 64 bit load/store instructions.

Interestingly, those two serialize calls don't panic (that doesn't necessarily mean that they didn't deserialize garbage, though):

grmtools/lrpar/src/lib/ctbuilder.rs

Lines 833 to 834 in f16a9d0

	let grm = deserialize(grm_buf).unwrap();
	let sgraph = deserialize(sgraph_buf).unwrap();

Adding a test that does a round-trip of serializing/deserializing the YaccGrammar<StorageT>, StateGraph<StorageT> and StateTable<StorageT> structs could identify the problem, I'll start with that.

[pablosichert]

Thinking about it – I'm pretty certain you can expose the same behavior if you just „regularly“ build for 32 bit.

I'm on macOS Catalina, which doesn't support 32 bit anymore, but I'll try to get a system running to validate that assumption. Might make debugging a lot easier.

[ltratt]

That suggests to me that the problem is that you're generating the grammar files on a 64-bit machine (OS X) and then trying to run them on a 32-bit machine (WASM). lrpar isn't set up for cross compilation at the moment, so that mix will be problematic. My guess is that if you can install a 32-bit OS (e.g. a Linux of some sorts) then this problem might go away.

[pablosichert]

Oh – spot-on!

I completely missed that build.rs is run on the host machine / host architecture.

Building the project in a 32 bit container worked.

Thank you!

[ltratt]

Cool! I also guess that means it works fine on 32-bit which is nice to know!

[clord]

I'm trying to build some software for a 32bit raspberry pi, and hit the same issue, but i can't run in a 32 bit host because rpi crosscompiler tools are all 64-bit. I might have to drop 32bit support, or pick another parser. how hard to make the build scripts be platform independent? perhaps encode __STABLE_DATA with json instead of binary so naked usize is not present?

[ltratt]

Can you use @pablosichert's hack of building your stuff in a 32 bit Linux container?

If not, my guess is that the right place to fiddle is with the use of the bincode crate -- probably https://docs.rs/bincode/1.3.1/bincode/config/index.html in particular. If you can work out what the right incantation for that is, I'm willing to look into formalising it in the API somewhere.

[There is a possible warning, though: from memory, some of your code will be type-checked by Rust with 64 bit ints, but you'll then move to 32 bit ints later. In practise, and possibly in theory, this probably won't cause any issues, but it would be remiss of me not to at least mention the possibility.]

[clord]

I tried the 32bit container, but rpi tools are all pre-compiled 64bit binaries, so no-go. I managed to get things working by using a 64bit container running 32bit multi-arch rust targeting arm. then the rpi cross-tools are happy, and 32bit rust cross-compiler produces the right output for arm32. it's a bit hacky :) but working. I still wish __STABLE_DATA could somehow be encoded in some better-defined type. usize is a bad type for serialization; would be better to define size, like u64.

[ltratt]

I'm glad you've got something working!