Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to define Links between plots #2832

Merged
merged 19 commits into from Jul 3, 2018

Conversation

Projects
None yet
3 participants
@philippjfr
Copy link
Contributor

philippjfr commented Jun 26, 2018

This PR adds the concept of a Link to HoloViews. Conceptually the Link is somewhat similar to a Stream in that it subscribes to changes and events on some source object and in turn triggers some event in response to it. However unlike a Stream it will not transmit that event from the frontend to the backend, instead it directly triggers some action on the target object. It therefore allows complex linked behaviors to be executed without involving a Python kernel.

This has several major benefits:

  • The linked action can be performed even in a static plot
  • The data does not have to be sent across a websocket (and then trigger an event which again sends stuff back across a websocket)
  • It is much faster and more responsive than python callback

The design/implementation of the Link also borrows very heavily from the inspiration of the Stream classes which has proven itself to be very solid and flexible. Each Link has a corresponding LinkCallback which defines what happens in case an event on the source is triggered. In the case of bokeh (currently the only supported backend for Links), the LinkCallback defines a (hopefully) small amount of JS which defines the action to be performed. In addition just like a StreamCallback it defines which models the callback should have access to and on which events/changes should trigger it. In this way we can concisely declare the linkage between two visualizations.

As a simple example I've prototyped a PathTableLink, what this does is show a table of the currently selected Path.

path = hv.Path([np.random.rand(10, 2), np.random.rand(10, 2)], ['xs', 'ys']).options(tools=['tap'])
table = hv.Table([], ['xs', 'ys'])
hv.links.PathTableLink(path, table)
path + table

path_table_link

  • Write documentation about links
  • Add at least 2-3 example Link classes along with reference examples
  • Add unit tests
@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jun 26, 2018

@jbednar @jlstevens Before I work on this much more thoughts and comments would be appreciated. While implementing linked tables and paths using streams I found that things were simply much too slow when editing large-ish paths/polygons. I've also been thinking about something like this for a while when looking at altair since they provide various ways of linking plots, e.g. see their interactive histogram: https://altair-viz.github.io/gallery/selection_histogram.html

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jun 27, 2018

Here is another example of a Link I just prototyped:

hist_link

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jun 27, 2018

It also provides a good mechanism to declare the new bokeh RangeTool:

%%opts Curve [shared_axes=False]
dates = pd.date_range('2015-01-01', '2017-09-26')
data = np.random.randn(1000).cumsum()
tgt = hv.Curve((dates, data), 'Date').options(width=800)
src = hv.Curve((dates, data), 'Date').options(width=800, height=100)
RangeLink(src, tgt)
(tgt + src).cols(1)

linked_ranges

@@ -0,0 +1,45 @@
from collections import defaultdict

This comment has been minimized.

@jlstevens

jlstevens Jun 28, 2018

Contributor

Couple of suggestions where this might live:

  • Maybe hv.util.links?
  • Or maybe hv.plotting.links (this might be better actually)

This comment has been minimized.

@philippjfr

philippjfr Jun 28, 2018

Author Contributor

Let's go with hv.plotting.links.

class PathTableLink(Link):
"""
Links the currently selected Path to a Table.
"""

This comment has been minimized.

@jlstevens

jlstevens Jun 28, 2018

Contributor

This is a rather specific example that is more of a specialized, custom link. Ideally we can find some very general link types that make sense to everybody.

This comment has been minimized.

@philippjfr

philippjfr Jun 28, 2018

Author Contributor

Yes, still thinking about a few examples, the RangeToolLink I demonstrated above is definitely one that will be generally useful.

@@ -355,6 +356,39 @@ def sync_sources(self):
self.handles['shared_sources'] = shared_sources
self.handles['source_cols'] = source_cols

def init_links(self):

This comment has been minimized.

@jlstevens

jlstevens Jun 28, 2018

Contributor

The three methods below are tightly coupled to each other but only loosely coupled to BokehPlot (only other use of self is self.traverse). For this reason I would consider moving them to one of the appropriate Link classes.

@jbednar

This comment has been minimized.

Copy link
Contributor

jbednar commented Jun 28, 2018

This looks fabulous, both for speed and for making things work in static exports! A user guide for how to write a Link class would be really helpful, because even though it's an advanced topic, it's something people would be very motivated to do so that they could make some amazing static plots to cover common cases in their domain.

Will it be possible to use Link and Stream together, so that plots are linked in static exports but some other (less crucial) Python-derived functionality is available when there is a live server?

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jun 28, 2018

Will it be possible to use Link and Stream together, so that plots are linked in static exports but some other (less crucial) Python-derived functionality is available when there is a live server?

Absolutely, e.g. take the Path-Table link, you could click on a path, it would then show the vertices in the table, and then you could subscribe to the currently shown vertices with a stream.

@jlstevens

This comment has been minimized.

Copy link
Contributor

jlstevens commented Jun 28, 2018

Overall I think this approach is fine as it isn't too different from what we do for streams. I'll know better once I see some docs with general examples: right now RangeLink is the most compelling demo (and I think it ought to be called RangeToolLink). Though, that does make it bokeh specific so maybe it should live in the bokeh plotting sub package?

@jlstevens

This comment has been minimized.

Copy link
Contributor

jlstevens commented Jun 28, 2018

One thing that would be good would be to ensure the docs talk to people familiar with BokehJS but not holoviews so the reader can easily apply their bokeh knowledge to building new types of link.

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jun 28, 2018

One thing that would be good would be to ensure the docs talk to people familiar with BokehJS

That's definitely sensible and at least one example should cover that. One thing I haven't quite made clear yet is that Links are not necessarily required to contain JS code, e.g. the RangeToolLink is pure Python and just adds the RangeTool to one plot and links it to the ranges on another. Similarly we had already discussed an explicit DataLink, which could also be trivially implemented using this approach without any CustomJS.

@jbednar

This comment has been minimized.

Copy link
Contributor

jbednar commented Jun 28, 2018

Excellent. Definitely need good docs to make this clear.

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jul 2, 2018

Okay, I've now added demo examples, a user guide and unit tests. I've decided against switching Links to ParameterizedFunctions because they can be called multiple times which would cause issues and they are more complex to explain to users.

This is ready to review now and can be polished further in later PRs.

"from holoviews.plotting.links import DataLink\n",
"\n",
"scatter1 = hv.Scatter(np.random.randn(1000, 2))\n",
"scatter2 = hv.Scatter(np.random.randn(1000, 2)*2, 'x2', 'y2')\n",

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

Would be good to use some structured data here to make the nature of the data link clearer. The data could be real data or artificial data structured in some way so you can understand how 'rows' are linked between the plots.

"metadata": {},
"source": [
"When working with the bokeh backend in HoloViews complex interactivity can be achieved using very little code, whether that is shared axes, which zoom and pan together or shared datasources, which allow for linked cross-filtering. Separately it is possible to create custom interactions by attaching LinkedStreams to a plot and thereby triggering events on interactions with the plot. The Streams based interactivity affords a lot of flexibility to declare custom interactivity on a plot, however it always requires a live Python kernel to be connected either via the notebook or bokeh server. The ``Link`` classes described in this user guide however allow declaring interactions which do not require a live server, opening up the possibility of declaring complex interactions in a plot.\n",
"\n",

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

... of declaring complex interactions in a plot that can be exported to a static HTML file?

This comment has been minimized.

@philippjfr

philippjfr Jul 2, 2018

Author Contributor

Sounds good.

"## What is a Link?\n",
"\n",
"A Link defines some connection between a source and target object in their visualization. It is quite similar to a Stream as it allows defining callbacks in response to some change or event on the source object, however, unlike a Stream, it does not transfer data and make it available to user defined subscribers. Instead a Link directly causes some action to occur on the target, for JS based backends this usually means that a corresponding JS callback will effect some change on the target in response to a change on the source.\n",
"\n",

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

... however, unlike a Stream, it does not transfer data between the browser and a Python process?

This comment has been minimized.

@philippjfr

philippjfr Jul 2, 2018

Author Contributor

Yep, seems clearer.

"A Link defines some connection between a source and target object in their visualization. It is quite similar to a Stream as it allows defining callbacks in response to some change or event on the source object, however, unlike a Stream, it does not transfer data and make it available to user defined subscribers. Instead a Link directly causes some action to occur on the target, for JS based backends this usually means that a corresponding JS callback will effect some change on the target in response to a change on the source.\n",
"\n",
"One of the simplest examples of a Link is the DataLink which links the data from two sources as long as they match in length, e.g. below we create two elements with data of the same length. By declaring a ``DataLink`` between the two we can ensure they are linked and can be selected together:"
]

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

It isn't clear to me how a DataLink extends beyond selections. Maybe another datalink example showing simultaneous (linked) updates would make it clearer?

This comment has been minimized.

@philippjfr

philippjfr Jul 2, 2018

Author Contributor

DataLink is primarily for linked selections and for slight efficiency gains, not sure what you mean by linked updates.

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

In that case I'm not sure how the data is linked. Sounds like the selections are linked (unless this achieves some other useful behavior). So SelectionLink?

"metadata": {},
"source": [
"In this case we are interested in the 'source' handle, but we still have to tell it which events should trigger the callback. Bokeh callbacks can be grouped into two types model property changes and and events. For more detail on these two types of callbacks see the [Bokeh user guide](https://bokeh.pydata.org/en/latest/docs/user_guide/interaction/callbacks.html#userguide-interaction-jscallbacks).\n",
"\n",

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

'into two types model property changes'? I suspect some punctuation is missing here...

This comment has been minimized.

@philippjfr

philippjfr Jul 2, 2018

Author Contributor

Thanks.

"cell_type": "markdown",
"metadata": {},
"source": [
"In this case we are interested in the 'source' handle, but we still have to tell it which events should trigger the callback. Bokeh callbacks can be grouped into two types model property changes and and events. For more detail on these two types of callbacks see the [Bokeh user guide](https://bokeh.pydata.org/en/latest/docs/user_guide/interaction/callbacks.html#userguide-interaction-jscallbacks).\n",

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

'and and events'

"cell_type": "markdown",
"metadata": {},
"source": [
"Now we have the ``Link`` class we need to write the implementation in the form of a ``LinkCallback``. A ``LinkCallback`` should declare the ``source_model`` we want to listen to events on and a ``target_model``, declaring which model should be altered in response. To find out which models we can attach the ``Link`` to we can create a ``Plot`` instance and look at the ``plot.handles``, e.g. here we create a ``ScatterPlot`` and can see it has a 'source', which represents the ``ColumnDataSource``."

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

Right after '... write the implementation in the form of a LinkCallback' I would point to the two types of bokeh callback with the link (html this time!) to the bokeh docs. This helps make it more concrete by linking to bokeh earlier.

This comment has been minimized.

@philippjfr

philippjfr Jul 2, 2018

Author Contributor

I agree, I was hoping to find a way to make the link clearer. I also wish bokeh docs had some more detailed section about the kind of events that are available.

"We now want to change the ``glyph``, which defines the position of the ``HLine``, so we declare the ``target_model`` as ``'glyph'``. Having defined both the source and target model and the events we can finally start writing the JS callback that should be triggered. To declare it we simply define the ``source_code`` class attribute. To understand how to write this code we need to understand how the source and target models we have declared can be referenced from within the callback.\n",
"\n",
"The ``source_model`` will be made available by prefixing it with ``source_``, while the target model is made available with the prefix ``target_``. This means that the ``ColumnDataSource`` on the ``source`` can be referenced as ``source_source``, while the glyph on the target can be referenced as ``target_glyph``.\n",
"\n",

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

source_source? Would formatting like source_source help?

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

Ah 'prefixing' so source_source?

This comment has been minimized.

@philippjfr

philippjfr Jul 2, 2018

Author Contributor

Yes, that helps. I've also considered renaming the handle to source -> cds, since source_cds would be much clearer, but I suspect someone is relying on that. So maybe we could alias it for now.

This comment has been minimized.

@philippjfr

philippjfr Jul 2, 2018

Author Contributor

I'll go ahead with this, source_source just looks stupid and confusing.

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

I think renaming it this way will look less confusing and will also be clearer to understand...

" target_model = 'glyph'\n",
" \n",
" source_code = \"\"\"\n",
" var inds = source_source.selected.indices;\n",

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

Ok, so source_source is a real thing. I think I'm getting the idea but I am finding this rather confusing at the moment.

This comment has been minimized.

@philippjfr

philippjfr Jul 2, 2018

Author Contributor

It's very similar to the way streams work, but I'll have another go at clarifying this.

"metadata": {},
"outputs": [],
"source": [
"Link._callbacks['bokeh'][MeanHLineLink] = MeanHLineCallback"

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

Would be nice to have a better API for registering these things together than using _callbacks directly like this.

This comment has been minimized.

@philippjfr

philippjfr Jul 2, 2018

Author Contributor

True, LinkedStreams should probably mirror that API since it's identical.

@jlstevens

This comment has been minimized.

Copy link
Contributor

jlstevens commented Jul 2, 2018

I've had a pass through the user guide. My main, high level comments:

  • Would be good to mark the section about writing your own links as an 'advanced' topic. This is not something I expect most people to do and people shouldn't be worried if they can't understand how to write their own links. The first half about usage should be friendly, showing people how to use the links we offer but the second half is more for people working at the developer level.

  • It might be nice to show how to write a Link without JS using the bokeh Python API (if possible!). One of the things you mentioned is you shouldn't have to write JS in all cases.

]
}
],
"metadata": {

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

Please clear out all this metadata...

This comment has been minimized.

@philippjfr

philippjfr Jul 3, 2018

Author Contributor

Done.

"cell_type": "markdown",
"metadata": {},
"source": [
"## Declare plot"

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

It is a neat example but it needs a sentence here to explain what is going on and what the user should try to see the linked ranges in action...

This comment has been minimized.

@philippjfr

philippjfr Jul 2, 2018

Author Contributor

Agreed, at some point someone will have to go through all the demos and add some explanation. But you're right that new examples should have some text.

This comment has been minimized.

@jlstevens

jlstevens Jul 2, 2018

Contributor

Same issue with timeseries_range_tool.ipynb...

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jul 2, 2018

Would be good to mark the section about writing your own links as an 'advanced' topic.

Agreed. I'll expand the section about usage a bit more and maybe think of a third Link example to include.

It might be nice to show how to write a Link without JS using the bokeh Python API

I'll try to think of one more, RangeTool and linking data were the two obvious cases but maybe an explicit RangeLink might also be useful. There is only so much you can do at the Python API level.

@jlstevens

This comment has been minimized.

Copy link
Contributor

jlstevens commented Jul 2, 2018

I'm generally happy with the design and docs though I do think some polish will be needed in a subsequent PR. Once the comments above are addressed, I am happy to see this PR merged with the understanding that more tweaks will follow.

One other suggestion to make the Link objects more useful was to add an unlink method to undo any previous linking behavior. That suggestions doesn't have to be added here but I do think it is a good idea.

@philippjfr philippjfr force-pushed the link_feature branch from ae0e558 to a8df15b Jul 3, 2018

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jul 3, 2018

Addressed most of the comments, merging now.

@philippjfr philippjfr merged commit 3cb7c83 into master Jul 3, 2018

4 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.08%) to 82.939%
Details
s3-reference-data-cache Test data is cached.
Details

@philippjfr philippjfr deleted the link_feature branch Jul 4, 2018

@philippjfr philippjfr added this to the v1.11.0 milestone Nov 5, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.