3D interactivity / picking

[sofroniewn]

🚀 Feature

Right now although many of our layers (Image, Labels, Points etc.) support 3D rendering, interactivity in 3D (i.e. painting, picking, adding points) has yet to be fully explored or supported.

[jni]

Thomas Robitaille showed some pretty neat 3D interactivity in glue at SciPy 2019:

https://youtu.be/TkMZ9gZ8xtk

Might be worth checking out that functionality and code base for inspiration.

There is also a much longer tutorial, though I haven't seen it yet:

https://youtu.be/8RkCpq9wMuI

[GenevieveBuckley]

FYI, Jan Arnold's 3D correlation toolbox (3DCT) does some 3D point picking. From memory there are a few nice things about the way it handles stuff, like fitting gaussians to bright blobs then "snapping" to them to fit the location more precisely.

It's GPLv3 (copyleft), the main repository is here: https://github.com/coraxx/3DCT

[jni]

GPLv3 means we theoretically can't even look at the code to port it without breaching the license terms. 🤦‍♂

[GenevieveBuckley]

We also discussed interactive 3D point picking in our last lab meeting, and have a feature request.

Most 3D point picking methods tend to rely on choosing the brightest voxel in the ray trace path. This can work ok for fluorescence microscopy data, but this works poorly for very large image volumes, and it also doesn't work well for other modalities or noisy images (like cryo-EM). So this approach shouldn't be the only supported method for 3D point picking.

An alternative my boss has used and liked in the past is an orthoslice view, where you can toggle on/off a semi-transparent volume rendering around it, and where the orthoslice plane positions can be used as constraints on the selected point positions.

If three orthoslice planes are "on", then this uniquely determines the position of a point in 3D space. If only one orthoslice plane is "on" we have the same behaviour napari currently has. And if two of the orthoslice planes are "on" selected points snap to the closest point of the intersecting line between the two planes.

Using a method like this involves a lot of switching back and forth between selecting points & adjusting the orthoslice planes slightly, so the switching can't be too fiddly or annoying.

[alisterburt]

just adding that at least for points this would work really nicely without ambiguity in the hybrid 2d-3d mode you showed in #839 - the requirement would be that the active layer be a 2d slice, point gets added at mouse location

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/question-about-accessing-volume-rendering-coordinates/50975/2

[alisterburt]

with #3037 3D interactivity is starting to become a reality - I thought it might be good to build a list of features we want in this issue...

From discussion with @kevinyamauchi

• point selection in 3D
• selecting labels in 3D (Add view ray and labels selection in 3D #3037)
• painting on arbitrary slices through 3D data (arbitrary plane rendering prototype #3023)
• click and drag to move an arbitrary slice along its normal vector
• click and drag to move points/labels along axes (in world or data coordinates) in 3D