Handling large images

[kushalkolar]

Currently it is not possible to display images by using a Texture from an array larger than [8192, 8192], which seems to be a WGPU limitation:

Uncaught WGPU error (Unknown):
InvalidDimension(LimitExceeded { dim: Y, given: 9000, limit: 8192 })

I see a few potential ways to get around this:

1. Create a LargeImage class that stitches together pygfx.Image
2. Within pygfx.Image, use a single grid geometry with tiled textures if that's possible?
3. Get the indices of the data that are currently visible in the canvas, if it is larger than 8192 in any dimension then subsample the Texture data. If it is a zoomed-in view that is under 8192 then update the Texture data with the full data.

Is there a better way?

[almarklein]

Interesting problem!

Possible solutions can be divided into two categories: we could put the image into multiple textures and position these such that it looks like a single one, or, we could use a "smart" solution that uploads data dynamically. The former is simpler to achieve, but will only allow larger images up to a certain point, because you'll run into the boundary of available GPU memory. Therefore my inclination is to go for the latter, which scales up to much larger datasets, opening up interesting use-cases.

So indeed something like your nr. 3. Perhaps using two textures: one showing a full-scale downsampled version, and one full-res positioned in the visible region.

Get the indices of the data that are currently visible in the canvas

We don't do anything like this yet, but it's certainly possible. Also see #81.

This is, I think, quite a bit of work to do right. So if users need this quickly, we can first implement a stricthing-approach ...

[almarklein]

To clarify this a bit:

• The easy solution is to create multiple images that you position so that they seem like one larger texture.
  • I think it would be possible to subclass Image, cut the image up into multiple texture's to store in the geometry, and then modify the shader function to produce one Shader object for each texture.
  • An easier solution is to just create multiple Image objects and position them side-by-side.
• The harder solution is using tiling and LOD. This solution is worth including in PyGfx IMO, but it's not something one can implement overnight.

[kushalkolar]

thanks!

cut the image up into multiple texture's to store in the geometry, and then modify the shader function to produce one Shader object for each texture

Would there be significant performance benefits to using this rather than the simpler solution of tiling Image objects? I'll try out the simpler solution in the coming weeks!

[almarklein]

Would there be significant performance benefits to using this rather than the simpler solution of tiling Image objects?

I think the performance benefit would be negligible.

[kushalkolar]

Multiple Image objects approach implemented here, performance and array size is enough for our use cases so far 😄 fastplotlib/fastplotlib#143

[kushalkolar]

Did WGPU change the Texture size limit in the latest release? Textures > 8192 in either direction appear to be working now. Tried this on 2 computers and a new environment with pygfx v0.1.13 and wgpu-py v0.9.4

from wgpu.gui.auto import WgpuCanvas, run
import pygfx as gfx
import numpy as np

canvas = WgpuCanvas()
renderer = gfx.renderers.WgpuRenderer(canvas)
scene = gfx.Scene()

im = np.random.rand(10_000, 10_000).astype(np.float32)

image = gfx.Image(
    gfx.Geometry(grid=gfx.Texture(im, dim=2)),
    gfx.ImageBasicMaterial(clim=(0, 1), map=gfx.cm.plasma),
)
scene.add(image)

camera = gfx.OrthographicCamera(10_000, 10_000)
camera.local.position = (5_000, 5_000, 0)
camera.local.scale_y = -1


if __name__ == "__main__":
    canvas.request_draw(lambda: renderer.render(scene, camera))
    run()

EDIT: I can confirm that with wgpu-py v0.8.4 and pygfx v0.1.10 this throws the original error from this issue. So I guess something changed recently 😄

Uncaught WGPU error (Unknown):
InvalidDimension(LimitExceeded { dim: X, given: 10000, limit: 8192 })

[almarklein]

These limits are defined by wgpu-native, which takes them from wgpu-types, which seems to query them from the underlying API (Vulkan/Metal/DX12), so it's hard to tell what changed.

[Korijn]

Did you update drivers, or OS, or hardware?