[suggestion] progressive loading for textures

[munrocket]

Since we in the web, we can introduce some API that will help us to fetch jpeg/webp from server progressively and bind it to GPU for us without lot of work.

[Kangz]

This is very similar to what game engines call "texture streaming". I'm not entirely sure how they do it, but one way that should be possible in WebGPU is by sampling textures with an explicit LOD that's clamped to the maximum level of the texture that's currently loaded.

However the details of how progressive loading would work is very application-dependent. For example do you want a smooth transition or not? Do you do progressive decoding like progressive PNGs where you can see the LOD line go down? What do you do if the texture stopped streaming? What format does the texture come in as? That's the first reason I think this shouldn't be in WebGPU but could be done by an opinionated library instead. (or maybe you can just use Basis?)

The other reason is that WebGPU gives low-level access to the GPU and pre-bakes a lot of objects for faster and predictable execution. Being able to change sampling parameters in the middle of commands already baked would be very complicated and would negate a lot of the advantages of WebGPU.

[munrocket]

Basis it's not tiny additional wasm blob, also it's not solving issue with progressive loading but trying to shrink images. In the end it uses existed API.

For example do you want a smooth transition or not?

Nobody doing that, just updating with new texture would be enough. Browsers already support this and this looks like routine operation. Doesn't know what progressive PNGs it is, never heard about it.

The reason why I open the issue here is because there might be some low level improvement in streaming this textures to GPU. Right now programmers need to create 2-4 version of a texture and always remove previous version from GPU to emulate this in software, all of this versions in JPG and all of them support progressive loading, that's why it looks like design issue. Offline applications never facing with this, they can just wait little bit, hard drive is faster than network.

[Kangz]

Basis it's not tiny additional wasm blob, also it's not solving issue with progressive loading but trying to shrink images. In the end it uses existed API.

For example do you want a smooth transition or not?

Nobody doing that, just updating with new texture would be enough. Browsers already support this and this looks like routine operation. Doesn't know what progressive PNGs it is, never heard about it.

Whoops, meant progressive JPEG, that's the image format that allows browsers to "stream" the texture. Also I've seen some games use the progressive LOD for texture streaming.

The reason why I open the issue here is because there might be some low level improvement in streaming this textures to GPU. Right now programmers need to create 2-4 version of a texture and always remove previous version from GPU to emulate this in software, all of this versions in JPG and all of them support progressive loading, that's why it looks like design issue. Offline applications never facing with this, they can just wait little bit, hard drive is faster than network.

I suggested a method above that doesn't require creating multiple versions of a texture.

My second point above was that integrating such a functionality in WebGPU would be extremely challenging and would introduce a lot of complexity and underlying implementation details that would make the whole API slower in general, even if you don't use the feature (because we have to prepare things like GPUBindGroup so we can change the LOD), more difficult to reason about for developers. These are against the goals of WebGPU.

HTML can do progressive loading because it is a format describing a document and the browser has large freedom to implement rendering however it wants. However WebGPU doesn't have as much freedom because it lets user control the GPU directly.

So I think there is space for a library that helps with this, or a tutorial explaining how progressive loading can be achieved, but imho it is not possible to integrate such functionality directly in WebGPU. I'll let others chime in.

[magcius]

It would be interesting to get access to the partial progressive pixels of a JPEG as it's being loaded in script, but that's probably a feature request for some other Web API and not directly related to WebGPU. I don't believe this is possible right now. However, it would make more sense to save off the mip levels as they come in, and I don't think the browser guarantees that every level in a progressive JPEG will be generated, nor is it guaranteed they'll be saved off.

Not to mention the lack of alpha channel, and the fact that this will decode to RGBA8888 and not something like BCn. Fine for small apps, I suppose, but most people likely want to ship some form of compressed texture for the VRAM savings. And so Basis is starting to not look like a bad option here.

[kainino0x]

It would be interesting to get access to the partial progressive pixels of a JPEG as it's being loaded in script, but that's probably a feature request for some other Web API and not directly related to WebGPU.

This is a really interesting request, but I agree this whole topic is definitely orthogonal to WebGPU.

Perhaps this is something Web Codecs would be interested in.

[munrocket]

I don't believe this is possible right now.

Doesn’t know that, thanks. It can be some sort of proposal in tc39. I will close this issue soon ( or you can close it by yourself as resolved earlier ).

[kvark]

@Kangz

one way that should be possible in WebGPU is by sampling textures with an explicit LOD that's clamped to the maximum level of the texture that's currently loaded.

I hope they don't do this, since that would require us to zero-initialize the mipmap levels they are never sampling from.

[Kangz]

@Kangz

one way that should be possible in WebGPU is by sampling textures with an explicit LOD that's clamped to the maximum level of the texture that's currently loaded.

I hope they don't do this, since that would require us to zero-initialize the mipmap levels they are never sampling from.

The need to allocate the levels anyway, so zero-initializing is fine imho.

[munrocket]

Trying to find a champion from TC39 to wrap this idea in a proposal and add this ideas to the ecmascript specification https://es.discourse.group/t/jpeg-progressive-loading-for-webgl-webgpu/359

Thanks for help.

[kainino0x]

TC39 is for JavaScript the language, not the web platform. I think it makes sense in WICG (e.g. under the Web Codecs group I mentioned, or separately).

[kainino0x]

Also note the ImageDecoder proposal you mentioned was also discussed at WICG.

[munrocket]

This is cool!