API for rendering canvas-visuals to an RGB image

[bgraedel]

🚀 Feature

Implement a method to render the canvas as an RGB image directly.

Motivation

There are often cases where I would want to export the canvas as an RGB image at its native resolution. The current screenshot option in napari is useful, but it requires the user to manually adjust the canvas or crop the images afterwards, which can be cumbersome and often not very fast. It is also not easily possible (and in some cases impossible) to export the canvas-visuals at their native resolution. By using the viewer.window.resize method this can be mitigated a bit, however this is constrained by minimal and maximal viewer size, which is related to scaling and dock-widgets and so on.

Pitch

I would like to propose the implementation of a method or an additional argument to the screenshot method, to directly render the canvas as an RGB image at the datas native resolution. I have experimented with some options all of which would require access to the canvas attribute, so implementing it as a plugin is not possible since the access to this will be removed in 0.5.0.

The simplest way I found to do this was to resize the canvas, then calling the canvas.render(), or viewer.window.screenshot(canvas_only=True) method and save the image as an RGB.

class CameraSetter:
    """A context manager to adjust viewer camera settings before rendering."""
    
    def __init__(self, viewer):
        self.viewer = viewer
        # get initial settings
        self.center = viewer.camera.center
        self.zoom = viewer.camera.zoom
        self.angles = viewer.camera.angles

        self.input_canvas_size = viewer.window.qt_viewer.canvas.size

        extent = viewer._sliced_extent_world[:, -2:]
        scene_size = (extent[1] - extent[0]) / viewer.window.qt_viewer.canvas.pixel_scale # adjust for pixel scaling
        grid_size = list(viewer.grid.actual_shape(len(viewer.layers)))
        
        # Adjust grid_size if necessary
        if len(scene_size) > len(grid_size):
            grid_size = [1] * (len(scene_size) - len(grid_size)) + grid_size
        
        # calculate target size i.e the size the canvas should be to fit the whole scene
        self.target_size = tuple((scene_size * grid_size[::-1]).astype(int))
        self.center = viewer.camera.center
        self.zoom = viewer.camera.zoom
        self.angles = viewer.camera.angles

    # copied from viewer.reset_view and modified without padding
    def _center_on_canvas(self):
        """Reset the camera view."""
        extent = self.viewer._sliced_extent_world
        scene_size = extent[1] - extent[0]
        corner = extent[0]
        grid_size = list(self.viewer.grid.actual_shape(len(self.viewer.layers)))
        if len(scene_size) > len(grid_size):
            grid_size = [1] * (len(scene_size) - len(grid_size)) + grid_size
        size = np.multiply(scene_size, grid_size)
        center = np.add(corner, np.divide(size, 2))[-self.viewer.dims.ndisplay :]
        center = [0] * (self.viewer.dims.ndisplay - len(center)) + list(center)
        self.viewer.camera.center = center

        if np.max(size) == 0:
            self.viewer.camera.zoom = np.min(self.viewer._canvas_size)
        else:
            scale = np.array(size[-2:])
            scale[np.isclose(scale, 0)] = 1
            self.viewer.camera.zoom = 1 * np.min(
                np.array(self.viewer._canvas_size) / scale
            )
        self.viewer.camera.angles = (0, 0, 90)

    def __enter__(self):
        """Set up the viewer for rendering."""
        self.viewer.window.qt_viewer.canvas.size = self.target_size
        self._center_on_canvas()

    def __exit__(self, exc_type, exc_val, exc_tb):
        """Reset the viewer after rendering."""
        self.viewer.window.qt_viewer.canvas.size = self.input_canvas_size
        self.viewer.camera.center = self.center
        self.viewer.camera.zoom = self.zoom
        self.viewer.camera.angles = self.angles

def render_as_rgb(viewer):
    """Render the viewer for a single timepoint."""
    with CameraSetter(viewer):
        rendered_img = viewer.window.qt_viewer.canvas.render(alpha=False)
    return rendered_img

# Usage
with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    rendered_img = render_as_rgb(viewer)

Of course this is a failrly naive implementation and one would have to think about how to handle 3d data, very large images, overlays and so on. But I think it could be extremely useful to have a feature like this (in any form what so ever).

Alternatives

• Stick with the current screenshot method
• canvas.render() would in theory take a region and size attribute so resizing the canvas could be skipped, however I did not manage to get this to work reliably, even when accounting for monitor scaling and canvas-to-world coordinate transforms. It also has issues if the camera was set so that no the entire image is visible on the canvas.
• Experiment a bit more with vispys lower-level function to find an approach e.g using gloo and fbo but i really dont understand enough of this to judge wheter this is easily possible.

[Czaki]

It looks like this Issue is partially related to #6114. Also qt_viewer deprecation was postponed to 0.6.0 in #6283.

In the context of Proposed API changes I prefer to allow provide required resolution instead of using "native" as it may lead to problematic results for big datasets.

[bgraedel]

Thank you for the feedback!

I've looked at #6114 and can see the overlap. My bad for not looking more thouroughly before... I think that resizing the canvas and reseting the camera would be more straight forward than cropping the black canvas. The approach above has worked pretty well for me so far (however maybe there are special cases which im not considering)

I guess since the deprecation was postponed its not very urgent however would still be great to have a feature like this.

I agree that having a size-argument would make sense, how about making this an optional argument and by-default export at the datas "native" resolution.

Would there be interested in a pull request that implements something like this (e.g a new method to the window class next to screenshot i.e. to_rgb or something like this) ? I could give it a try. Or is there already somebody working on a feature like this or a more sophisticated version? (I saw something mentioned here https://forum.image.sc/t/saving-image-from-napari/50379/30)

[Czaki]

I agree that having a size-argument would make sense, how about making this an optional argument and by-default export at the datas "native" resolution.

what did you mean by native resolution for 3d rendering? For 2d it is clear but not for 3d.

[bgraedel]

hmm so the way it is set up above it would reset the camera angle to the default state i.e (0, 0, 90). so "native" in this case would probably mean yx extent, but yes it is much less clear for the 3d case

[aeisenbarth]

Such an API is hugely desired from many sides.

It would be nice being able to use the API headless, without launching the Napari GUI, and if possible avoid depending on Qt (just render the image as array).

This discussion has some overlap with spatialdata-plot, which implements rendering specifically for the SpatialData format.

Also, such an API should keep in mind the link between data and the rendering process. Some data formats store visualization parameter presets. Developers read these presets and pass them to a visualization API (like this). It would be good to coordinate naming of rendering parameters between data formats and implementors of visualization tools. As example, NGFF-Zarr stores OMERO rendering parameters, which resemble quite a lot Napari's add_image parameters (color→colormap, window→contrast limits).

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/retrieving-burned-displayed-image-data-in-napari/92149/2