Memory64 support for Wasm with large (>4GB) initial data segments

[ryankalla]

Hello

We have a production use case where we would like to use Wasmtime to run a Wasm module that has been initialized with more than 4GB of data. I have been running into a few u32 overflows while trying this (even when the memory64 config setting is enabled). Some of the errors seem to be coming from the "wasm-encoder" and "wasmparser" crates, and some from the Wasmtime code itself. Specifically the errors seems to be happening when Wasmtime is compiling and initializing the Wasm module before execution (see links below).

Note, this doesn't seem to be an issue when the Wasm module starts with less than 4GB of data and then grows to more than 4GB of data while running. Only if the Wasm module starts with more than 4GB of data.

I was able to get things working in a fork by changing those u32s to u64 (or usize in some cases). I was wondering if there are plans to add support for this use case? Or maybe I have missed something about why it is implemented as is?

Here's a link to my Wasmtime diff for reference: main...ryankalla:wasmtime:main

And here is my Wasm-Tools diff: bytecodealliance/wasm-tools@main...ryankalla:wasm-tools:main

P.S.
I used Wizer to initialize and snapshot the Wasm module I'm using (this is where I'm loading >4GB of data into memory). I needed to make some similar changes to Wizer as well to enable memory64 and support these large data sizes.

Here is a link to my Wizer diff: bytecodealliance/wizer@main...ryankalla:wizer:main

Let me know if I should open separate issues in the Wasm-Tools and Wizer repos, or just keep the whole discussion here.

Thanks!

[alexcrichton]

Thanks for the report, and this is an interesting use case! I think the answer here is you'll probably want to skip Wizer for now since supporting modules >4G is likely going to involve standardization work.

• Technically wasmparser is being more permissive than other engines by validating this large of a module at all. The specification limits indicate that modules need to be <1G which neither wasmparser nor wasmtime currently enforces. We otherwise try to keep the limits in sync with other engines (including web engines) to ensure that if a module runs in one location it runs everywhere. Given that this technically should blow the limit just because of the size. That being said there's no strict reason we need to precisely follow the exact same limits, so this isn't a hard blocker.
• The current wasm binary encoding specifies the length of sections as a 32-bit integer so it's impossible for any one section to be larger than 4G. Your changes to wasm-tools, for example, are invalid and change the binary encoding/meaning of certain sections. What you've sketched out of course works, but that's a change that will need to be proposed at the standards level as opposed to just changing things in wasmparser.

For now I think it might be best to update Wizer to fail if the memory snapshot is >4G. That's about the best that can be done today since the memory snapshot in general can't be larger than that due to wasm encoding limitations. At the standards level the "easiest" solution might be to change section lengths to a 64-bit integer instead of a 32-bit integer and then to chunk the image in 4G data segments (as opposed to letting data segments get larger than 4G). Even that though is likely to have some lively discussion in the CG, so would take awhile to prototype and get agreement on.

[ryankalla]

Hi Alex, thanks for the quick reply!

Good to know about the limits specified in the JS API. Glad to hear that it's not a blocker to have more permissive size limits in non-browser environments.

Just to confirm, is this the part of the spec you're referring to about 32-bit section lengths? Offhand do you know the rationale for keeping that at 32-bits with Memory64 becoming standard?

[alexcrichton]

That's one big one yeah, and the other is that the generic encoding of list(T) in the wasm spec uses a 32-bit length as opposed to a 64-bit length. Where you linked means that a single section can't be larger than 4G, and the 32-bit list length means that a single data segment also can't be larger than 4G. Coupled with how you can't have multiple data sections, it means that the maximum data segment is just shy of 4G in size.

Memory64 in general mostly dealt with the runtime side of things and things feeding into that. AFAIK binary encoding wasn't intentionally ignored it just probably never came up. With web engines still limiting the size of a module to 1G it's also not relevant for any web engines either. I suspect the 1G limitation on the total size of a module is why this has never come up in a web context or why many other folks haven't run into it.

[alexcrichton]

I'm going to close this since there's not much we can do in Wasmtime to resolve this and it's mostly an upstream wasm CG issue.