Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Napari for large volumetric data; feedback and issues #471

Open
constantinpape opened this issue Aug 9, 2019 · 3 comments
Open

Napari for large volumetric data; feedback and issues #471

constantinpape opened this issue Aug 9, 2019 · 3 comments
Labels
Milestone

Comments

@constantinpape
Copy link

@constantinpape constantinpape commented Aug 9, 2019

I have been exploring the use of napari for displaying large volumetric data and wanted to provide some issues / feedback.
Before that: thanks a lot for developing napari, I really like the design of the API and GUI and it's definitely the best python viewer I know already.

Using napari for large volumetric data:

  • Performance: I have been using napari with larger-than-memory volumes by passing h5py or z5py (my own library for reading zarr and n5 files, I/O performance should be very similar to zarr-python) datasets to add_image or add_pyramid. For large datasets, the viewer is not very usable; scrolling to a new slice takes a few seconds and does not feel smooth. I can see the following issues / proposals:
    • Lazy rendering: Scroll to a new slice immediately and lazily load the slices / render the tiles (data that is not loaded will just be background color). This is done by BigDataViewer and improves the experience of scrolling through large datasets a lot. Related: #318
    • Cache tiles and/or data: performance could potentially be increased by caching tiles and/or data chunks (see below).
    • Access pattern matching chunks: Most large data sources are chunked and the viewer performance can potentially be increased if the data access is somehow aware of the chunk sizes.
  • Labels: Some general and big data specific feedback on the labels layer:
    • Immutable labels: the labels layer is very useful for displaying segmentation results. There are use-cases, where these should be static, i.e. the user should not be a able to paint.
    • Pyramid labels: In order to display segmentation results for large data, a LabelPyramidLayer would be necessary. Note that there are different approaches of storing the data for this purpose, the easiest being down-sampling by nearest neighbor interpolation, but also more complicated approaches like label multi-sets. Also, painting for mult-scale labels is a bit more complicated, so implementing a immutable version of this would be easier.
  • Usage: In general, I really like the napari GUI and usage, however some adjustments for volumetric data would be desirable:
    • Orthoviews: Enable the 3 orthoviews: xy, xz, yz. Related: #440, #451
    • Scroll with the mouse wheel: move zoom to Shift + Wheel or similar. Ideally, there would be add_mousebindings similar to add_keybindings to make the mouse behaviour configurable. Related: #252
    • Arbitrary reslicing: rotate the view angles around a central point, as supported by BigDataViewer. Needs support for affine transformations (see below). Related: #320
  • Support for affine transformations:
    • To support arbitrary reslicing.
    • To map different registrations / alignments of a dataset to the same world coordinates (as supported by BigDataViewer through ViewSetups).

I haven't spend much time peaking under the hood of napari, so please excuse if something I have listed here is implemented already and be so kind to point me to it.
If you think any of the points here is worth a separate issue, I am happy to open this with a bit more description.

Also, I have been working on convenience functions for napari and @jni suggested merging this into napari directly, see constantinpape/heimdall#5.
I am very open to this, however I think we should discuss the points I brought up here first.

Edit: I had a closer look at the issues here and referenced related issues.
And to add some context, I mostly work with EM data showing dense structures, so while the data is volumetric, volume rendering (= add_volume) is not very useful, hence this all refers to add_image for 3 dimensional data, but I imagine some of these points would apply to volume rendering as well.

@sofroniewn

This comment has been minimized.

Copy link
Contributor

@sofroniewn sofroniewn commented Aug 9, 2019

@constantinpape this list is amazing - that's so great you're already finding value from napari and building on top of it. I'd also love to see anything you're doing of general interest make back upstream into napari, but agree we can approach that gradually. It seems that you're further along on the advanced fileIO than we are - we've just started discussing it here #379.

My overall take though is that each of your major bullets is something that we really want to see napari be able to do. We want to make sure napari scales to enormous datasets, such as the EM ones you're using.

Looking though the list my hunch is that the Lazy rendering and Pyramid labels might be the most important features to you - is that the case? Let's keep the discussion going here for a bit to figure out how to prioritize your needs and how they fit in with what we're working on right now - which you can get a sense of here in features to 0.2. You can also get a sense of features we're got planned for a little bit further down the road - features to 0.3.

@constantinpape

This comment has been minimized.

Copy link
Author

@constantinpape constantinpape commented Aug 9, 2019

It seems that you're further along on the advanced fileIO than we are - we've just started discussing it here #379.

Thanks for referencing this. I will have a look later and maybe add some comments.

My overall take though is that each of your major bullets is something that we really want to see napari be able to do. We want to make sure napari scales to enormous datasets, such as the EM ones you're using.

Very happy to hear this :).

Looking though the list my hunch is that the Lazy rendering and Pyramid labels might be the most important features to you - is that the case?

That's pretty accurate. I would prioritize my needs as follows:

  1. Performance: Here, lazy rendering is indeed the most important point, because it needs to be implemented in napari. For caching, I can start by inserting an intermediate cache myself to gain first experiences with this.
  2. PyramidLabels: In order to view large segmentation results.
  3. Scrolling via mouse wheel / configurable mouse: For quick data navigation this is pretty essential.

All the rest would be nice to have, but for now is not essential for me.

Let's keep the discussion going here for a bit to figure out how to prioritize your needs and how they fit in with what we're working on right now - which you can get a sense of here in features to 0.2. You can also get a sense of features we're got planned for a little bit further down the road - features to 0.3.

Sounds good! I will have a look at the feature issues later.

@sofroniewn

This comment has been minimized.

Copy link
Contributor

@sofroniewn sofroniewn commented Aug 9, 2019

Nice.

  1. For lazy rendering if you were interested in contributing this feature to napari that would be amazing, we could help you with the integration. Probably worth continuing a discussion about this feature specifically on #318 where i'm curious to get an update from @d-v-b too. It might require some changes on the vispy end of things and we'll have to think more about the details, but a couple very concrete examples will be helpful.

  2. PyramidLabels is straight forward in some sense, but I think we want to do it in a way that reduces duplicate code. @jni and I had a little discussion on a recent PR about combining some of the image / pyramid / labels stuff to reduce duplication and I want to think more about that before just copying the current stuff, but this is 100% something we will support.

  3. Scrolling with the mouse should be exposable with Qt, I can look into that. For the custom mouse functions if you could add a note to #252 about exactly what functionality you'd like / maybe an example use case that would be great as we've been having some discussion lately about exactly how best to support this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.