Avoid reading the entire asset into memory during asset processing.

[andriyDev]

Objective

• When processing assets, we first read the whole asset into memory, then process the asset from that in-memory representation. This means that large assets may just not fit into memory causing much bigger issues (e.g., switching over to slow virtual memory).

Solution

• Read the asset twice during processing: 1) once to determine the asset hash, 2) once to actually process the asset.

This means the processing code itself doesn't need to read the whole asset into memory at any point, meaning we can now process much bigger assets.

However, there are some risks. Asset sources which can't read chunks of an asset - which need to read the whole asset into memory anyway - now have to do so twice (not at the same time though). An example of this kind of asset source is the default Wasm source, or the HTTP asset source. In practice, I don't think this is a big issue - processing is likely to be happening on local assets anyway - it seems unlikely that users will want to download large assets from an HTTP asset source multiple times.

Another risk is that for asset processing, copying files is possibly slower - previously we just read the whole asset in, and then wrote the whole asset out. Now, we read small 1k chunks from the file, then write that chunk. This introduces some scheduling overhead in the copying. We can tune this number if we find it's too slow.

Testing

• The processing tests all still pass.

[JMS55]

Nice step towards https://zeux.io/2025/09/30/billions-of-triangles-in-minutes/ !

Some questions:

• What is the hash used for? And the duplicate reads is only for asset processing, and won't have any effect on release versions of games loading already-processed assets, right?
• We still don't have a way to read a certain slice of bytes from the asset, right? E.g. parsing a GLTF file to find meshes in them, then reading the meshes one at a time from other parts of the file

[andriyDev]

@JMS55 Yup haha, I think you posted this a while ago in the Discord and it's been rattling around in my brain as something we simply can't do with Bevy today. I intend to fix that!

• The hash is used when the processor initializes to determine whether an asset is up-to-date.
• You are correct, the duplicate reads only matter during asset processing. This has no effect for loading already-processed assets.
• Sorta. Currently Reader only requires AsyncRead and AsyncSeekForward. In theory, you could write a glTF loader that cleverly seeked in such a way that they never go backwards. That would be very painful though. So what I've been working on is to allow the loader to request features from the asset source. So a glTF loader could request the "seek from start" feature, and then its reader could do whatever seeks it needs - much simpler to work with. We'll see how it pans out!

[JMS55]

Looks pretty good to me, some last feedback.

[JMS55]

Can we add a nice helper for this?

[andriyDev]

I intentionally don't want to. We should be encouraging users to engage with the "buffered" API of reading and writing, to reduce how much memory we're using. Today a lot of our loaders and sources just read everything into memory, and that can really limit how much data we can process. Besides, this is exactly how users should be doing it in loaders anyway, so this is no different. This migration is just weird because the previous behavior of having all the bytes is weird.

[JMS55]

Suggested change

	// This means we've reached EOF, so we're done consume asset bytes.
	// This means we've reached EOF, so we're done consuming asset bytes.

[andriyDev]

Done! I also realized this was slightly wrong - it should be if we didn't fill the buffer that it keeps processing.

[JMS55]

Approved with some minor doc suggestions.