[WIP] 1D PlotWidget

[tlambert03]

Description

This is a branch from #675 that pulls the core PlotWidget functionality out of the histogram PR. I haven't yet explicitly vendored the vispy plotwidget, and may just stick with the relatively minimal implementation for this PR.

The main additions here are the NapariPlotWidget (my subclass of the vispy.plot.PlotWidget with modified initialization and napari-specific styles), and a QtPlotWidget (a QWidget with a vispy.SceneCanvas and a NapariPlotWidget in the main view.`
There is also a QLineProfile subclass and example. Histogram-specific stuff stays in #675, and I may additionally add a generic barplot to this PR, that I would use in the histogram PR.

(I tried to cleanup by squashing my commits in this PR, but kept running into conflicts, so we'll just do the squash eventually when merging).

Type of change

• New feature (non-breaking change which adds functionality)

How has this been tested?

• the test suite for my feature covers cases x, y, and z
• all tests pass with my change

Final checklist:

• My PR is the minimum possible work for the desired functionality
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[tlambert03]

ok, I reduced the code here significantly to what feels like a better starting place. Really, it's a very thin layer on top of the existing vispy.PlotWidget and Fig stuff. But as discussed before, there are enough parameters that aren't immediately accessible in the stock classes from vispy that I think it's best to have our own subclasses for a little bit, and we can also offer up any improvements back to vispy over time.

Usage basically looks like this at the moment:

• napari._vispy.Fig is basically just a vispy.plot.Fig. At its core, this is a vispy canvas, with a vispy Grid as its central widget. To create subplots (and grab their handles), you just index into the figure instance like this:

  fig = napari._vispy.Fig()
  ax0 = fig[0, 0]  # creates an empty PlotWidget in the first row, first column
  ax1 = fig[1, 0]  # creates an empty PlotWidget in the second row, first column

• napari._vispy.PlotWidget is a subclass of vispy.plot.PlotWidget but with a fair amount of overrides from the parent (here's the part where we can eventually push changes back to vispy). This widget contains the axes, ticks, labels, etc... for each subplot in the figure. It also has most of the interesting plotting methods (currently just plot and scatter... but this is where we can add more convenience methods). continuing on with the example above:

  # make a line plot with some X,Y data
  ax0.plot(X, Y, color='m', marker_size=0)
  # make a scatter plot with some data
  ax1.scatter(np.array((X, Y)).T, face_color='m')
  # etc...

• a quick way to make a new docked figure is to use viewer.window.add_docked_figure, which returns a tuple (figure, dock_widget), to which you can start adding subplots and stuff.

I updated examples/mouse_drag_callback_lineprofile.py, this time with multiple lines, each of which updates a particular subplot. This multi plot figure is closer to something that might be useful for @cudmore's applications:

and below is the output from examples/colocalization.py, showing both scatter and plot used on the same axis:

let me know what you think of this general pattern. If you like it, we can probably add a few more convenience functions to either create figures, or link their data to particular layers.

[sofroniewn]

@tlambert03 that comment was great timing, I've just been going through this now. First off, this is incredible functionality and 100% we want to support all of it in napari in as easy way as possible.

My main concern right now is that all this code is on the vispy / Qt side of the fence (i.e. on the view side) and non of it is one the model side. Very early on the napari code base was much more like this, and then under the guidance of @royerloic we separated out our model files which are vispy / Qt independent and our view files which depend on vispy and Qt and then used our event system to connect them. I think we've all found this a really great move that has made development and testing more robust and put us in a position where if we do decided to change our view technology (vispy / openGl / Qt etc) we'll be in a much better place for it.

I think we'll want to do the same thing here, and that most of these files are great starting points for view files, but we'll want to add the model files before this gets merged. We probably only need one model file for the "plot", but we want to think about what that looks like and what the API to interact with it.

From an api standpoint separating out the model and the view here will also mean that our users don't have to make calls through viewer.window too which we might want to make private to reinforce this model view separation (note that under the hood we'll still make that call, and plugin developers who want to access more functionality, add their own highly custom Qt stuff might want to make that call, but I don't think it should be required for basic functionality that napari provides).

So probably the most important thing for us all to agree on before going further is what does the user facing API look like and what does it do. I also wonder how these things interact with our current concept of layers. One option is to treat them like layers in some ways, but not others. For example, we could have calls like

layer = viewer.add_plot(data, ......)
layer = viewer.add_scatter(data, ....)
layer = viewer.add_hist(data, ....)

These layers would get added to our layers list, so they would be accessible in viewer.layers but they'd be assigned to a different canvas / figure, with a different camera, so they wouldn't be overlaid on our image like layers.

Another option is that they get added to viewer.1D_layers or some other name, which keeps them more separated from our normal layers, but I'm not sure I like that. We're going to need to think about having different layers assigned to different cameras / canvases in #760 anyway, and I'd like this to use the same machinery / approach (which might mean changing our current canvas object to look more like the dock figure, see #855.

I'm definitely up for focusing on this for our 0.2.11 release (assuming no breaking changes) or making this part of the first 0.3.0 release if we have breaking changes. I think a lot of people are going to be really excited about this functionality!!!

[tlambert03]

great points, thanks for the thorough thoughts!

I'm all for adding a model to this. but will have to think about it (and welcome any/all suggestions there too). In one sense, most of this code is "view"-like because that's all these things really are. The linescan is just a view of the data in the image layer, along the coordinates of some shape layer. The scatter plot is a view into the pairwise image data of two layers.

one thing that I would want to make sure we avoid with something like viewer.add_plot(data) is data duplication/desynchronization. If the data being visualized comes from one of the layers, then it feels like our potential model should just capture the nature of that link between the layers, rather than having its own plot.data that could potentially become out of sync with the layer (i.e. there should only be one "source of truth")...

of course, some users will want to plot data that is unrelated (or arbitrarily related) to a napari layer... but that's such a wide open use case that it's not clear to me what our model would be (other than the plot.data coming in).

if you (or anyone) has time to think about this, maybe some more concrete examples of what you want put in a plot/scatter model would help. Maybe you mean things like marker size? color? range/domain, etc? I can go in that direction if that's what you're envisioning.

[sofroniewn]

one thing that I would want to make sure we avoid with something like viewer.add_plot(data) is data duplication/desynchronization. If the data being visualized comes from one of the layers, then it feels like our potential model should just capture the nature of that link between the layers

absolutely, agreed - that is the very interesting aspect of this feature addition. i'll need to think on it more, but philosophically we're aligned

some users will want to plot data that is unrelated (or arbitrarily related) to a napari layer... but that's such a wide open use case that it's not clear to me what our model would be (other than the plot.data coming in)

I think that's fine, the point of the model is to just capture all the those free parameters / keyword args that the users might want to set (like colors, symbols etc) in the absence of the code and objects that actually render them so we are independent of the actual rendering machinery, but have all the necessary information.

Maybe you mean things like marker size? color? range/domain, etc?

Yes that would be important stuff if we want the user to be able to control those things (I could imagine saying that axes limits will be autogenerated at first, but people always want to control those things eventually)

if you (or anyone) has time to think about this, maybe some more concrete examples of what you want put in a plot/scatter model would help.

I can work on this. The only bit I don't have a clear sense on right now is how to handle the linking of data.

Do @royerloic and @jni have ideas?

[royerloic]

one thing that I would want to make sure we avoid with something like viewer.add_plot(data) is data duplication/desynchronization. If the data being visualized comes from one of the layers, then it feels like our potential model should just capture the nature of that link between the layers

absolutely, agreed - that is the very interesting aspect of this feature addition. i'll need to think on it more, but philosophically we're aligned

Indeed, we should think about what it means to have a 'tool' (like a profile plotting machinery) be linked to the data of two layers (image layer and shape layer). if we define a model object for the notion of a line profile, then the view could be in a widget (or something else). However, I think we need to step back and think about the bigger picture and how we could organise the concepts here:

Right now we have a clear idea of what layers are and how this works, but we don't have a good framework yet for thinking about 'other stuff' hanging around that... One approach is to define different types for 'what-is-not-a-layer'. A first prime candidate for that is 'live-analysis-tools-that-result-in-plots', and we could simply say that a viewer can have multiple plots (handled in a list similarly to layers, but with no explicit list display, just dockable widgets, list perhaps in menubar under windows or tools). Plots are linked to certain layers, and have a piece of code that knows how to update the plot based on how the layers that they depend on change... ideally we would have an abstract class for live-tools that handles most of that, and concrete classes for different kinds of live-tools such as line profiles etc... Other such 'live-analysis-tools' could be: return the frequency spectrum of an image, histogram, etc...

Live-analysis tools are not limited to images: another example: In case of layers such as shapes or labels, a possible live-tool would simply return statistics about the number of shapes of each kind, mean sizes, lengths, number of connected components... etc.. The widgets (views) could be tables etc... Importantly here, because we have a model/view separation, you could access the result of such an analysis from the api -- even if there is no view at all ! (headless).

Explicit model-level linkage between live-tools and layers would make it possible to warn the user when they try to delete a layer that is used by a tool, etc... (ideally without a popup please :-) )

We could later extend that to other live-tools that do not result in a plot, but perhaps in other types of outputs that would require arbitrary qt interfaces. In that case we have to have as much machinery in the base class as possible, and have such tools 'declare' what they do. At first I see only core-devs writing such tools, but as the pattern get refined, I can imagine that this could be the basis for plugins that offer simple read-only type analysis on whatever is in the layer list...

This discussion is starting to connect more and more to the notion of plugin... etc... The way i look at it: "good design in napari makes everything pluggable". Anybody can write a layer (well almost everybody), anybody could write a 'live-tool'.

Right now live tools should be read-only with respect to the layers. Creating new layers or modifying layers (am a bit against modifying layers, preference towards adding modified layers) would be reserved to 'functions' (functional plugins), or even 'arbitrary plugins' that could do more
-- however, we want to limit that as much as possible and have a rich plugin declarative vocabulary to be able to capture as much as possible...

In summary: live-analysis-tools are pieces of code that are linked to certain layers at creation and know how to compute a 'result' of which one obvious type is data that can be displayed as a plot (bar chart, scatter plot, line, ...). such tools are defined by that code, return a standardised representation, and recommend a type of view for it (plot widget). Ideally, this specification does not involve any qt code -- instead, that qt code is 'automatically' generated/selected based on what is declared: "I can compute an histogram from an image layer with this code and I would like it to be displayed as a bar chart, thank you." But of course, it could be something else entirely and not a histogram but a bar chart representing something else about the image... Once we have that generic machinery, we can start to do a lot of cool things.

The key here is to have the client-code as much as possible 'declarative' and not 'imperative'...

some users will want to plot data that is unrelated (or arbitrarily related) to a napari layer... but that's such a wide open use case that it's not clear to me what our model would be (other than the plot.data coming in)

the qt widgets for plotting are a resource that should be self-contained and modular that would be usable independently of what I described above. They could be used inside of more complex ( and arbitrarily 'rich' ) plugins but can be also standard views for the result of live-analysis-tools...

I think that's fine, the point of the model is to just capture all the those free parameters / keyword args that the users might want to set (like colors, symbols etc) in the absence of the code and objects that actually render them so we are independent of the actual rendering machinery, but have all the necessary information.

Maybe you mean things like marker size? color? range/domain, etc?

Yes that would be important stuff if we want the user to be able to control those things (I could imagine saying that axes limits will be autogenerated at first, but people always want to control those things eventually)

if you (or anyone) has time to think about this, maybe some more concrete examples of what you want put in a plot/scatter model would help.

I can work on this. The only bit I don't have a clear sense on right now is how to handle the linking of data.

We need some framework (perhaps as described above), and have a typology and vocabulary for how to organise things like line plots, histograms and other such live analysis.

I keep reusing the term 'live-analysis-tools' because i think this captures nicely a large category of such widgets that we would want to have in napari...

Do @royerloic and @jni have ideas?
above :-)

[sofroniewn]

@royerloic these are great comments and I think live-analysis-tools-that-result-in-plots will be a huge value add for napari. I see three things at play here

• 1D plots: scatter / line / hist
• live-analysis-tools-that-result-in-plots
• live-analysis-tools-that-result-in-other-qt-elements

The more I've thought about this the more I've come to thinking that we want to define a really nice interface for adding and overlaying 1D plots using the model/view paradigm we already have and add_* methods. We can take a lot of API inspiration from the refactor @kevinyamauchi is proposing in #858 - which is basically the same API for a scatter plot but which will result in a 2D/3D layer rather than a 1D layer.

I think we can then think about an event based / callback based system for linking data to enable the live-analysis functionality, but it will now be built on a firm, reusable foundation

[jni]

I just want to caution here against reinventing too many wheels... We originally thought about using PyQtGraph, which by my understanding implements a lot of this stuff, and I'd consider revisiting the decision not to use it.

Overall, once you get into semantic plotting, you are opening a giant can of worms. The talk here is starting to sound like Bokeh but on an OpenGL canvas and with more features, and that represents many human-years of very hard work.

In short, to mirror the discussion we're having on point annotations, if we are going to start dealing with 1D data sources linked to our nD data, and become a plotting library, (a) we should depend on pandas, but (b) maybe we shouldn't. =P

I don't know. I see that this functionality is super great, but I'm concerned that, if we want this to be truly generic, then maybe it is not something that should live in core napari.

[royerloic]

I just want to caution here against reinventing too many wheels... We originally thought about using PyQtGraph, which by my understanding implements a lot of this stuff, and I'd consider revisiting the decision not to use it.

Overall, once you get into semantic plotting, you are opening a giant can of worms. The talk here is starting to sound like Bokeh but on an OpenGL canvas and with more features, and that represents many human-years of very hard work.

In short, to mirror the discussion we're having on point annotations, if we are going to start dealing with 1D data sources linked to our nD data, and become a plotting library, (a) we should depend on pandas, but (b) maybe we shouldn't. =P

I don't know. I see that this functionality is super great, but I'm concerned that, if we want this to be truly generic, then maybe it is not something that should live in core napari.

I think there is some misunderstanding here.
I agree we don't want to recreate a plotting lib.
that's why the number of plotting types will be extremely limited and constrained,
and that's perfectly fine....

[sofroniewn]

I agree we don't want to recreate a plotting lib.
that's why the number of plotting types will be extremely limited and constrained,

While I agree with this on some level - I do think we want to add support for the basic plot types line, scatter, hist with a data properties that can be updated, but otherwise stay away from complex linking features. The reason for this is that we need to use some of these things internally - many people are asking for a histogram for example, and so it is little additional effort to expose those three plot types in a nice way for users to customize.

Also, as this PR demonstrates vispy has a lot of the foundational plotting abilities required for the three basic types I mention, and I'd rather add improvements to vispy to get it where it needs to be than add a big dependency like PyQtGraph - which might have more advanced linking / updating options, but as I said, I don't think we want to bite off too much of the linking stuff right now.

I think fully generic linking is a big driver of complexity and value add for projects like https://glueviz.org/ and is something we might not want in core (you could always do linking like is done in this PR, say directly overwriting the data property, which will allow people to make these sorts of utilities easily)

Overall, once you get into semantic plotting, you are opening a giant can of worms. The talk here is starting to sound like Bokeh but on an OpenGL canvas and with more features, and that represents many human-years of very hard work.

Yup absolutely wary of this, again, I think the linking aspect of this scares me the most, and maybe that should be left for advanced users and plugin developers to do, but I think providing a basic API for line, scatter, hist isn't too big an ask given where vispy is, and we can make the approach and API match what is happening in #858 - which is basically identical to scatter.

Anyway, looking forward to discussing this and #858 more. Curious what @kevinyamauchi thinks and if @tlambert03 wants to weigh in again now that some other people have commented? We can do a deeper dive on this on Thursday too

[kevinyamauchi]

@tlambert03 this looks awesome!

@sofroniewn I’m on the road right now, but I have time tomorrow to take a closer look. I’ll try to add my thoughts before tomorrow’s dev meeting!

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/how-to-add-plot-with-same-colour-as-label-layer/38453/2

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/interactive-plotting-in-napari/57112/1