Making widgets talk to each other! #86

Open
ramnathv opened this Issue Mar 6, 2015 · 58 comments

Projects

None yet
@ramnathv
Owner
ramnathv commented Mar 6, 2015

I have been experimenting with the ability to use widgets as components and then allow them to communicate with each other.

Here are some examples

  1. Life Expectancy Choropleth + Line
  2. Datamaps + Chroniton
  3. Datamaps + Morris + Sortable

The first example uses shiny, while the second does not.

In the case of Shiny, I am using Shiny.onInputChange to trigger changes and communicate back to the server from the client on mouseover. In the second example, I am directly manipulating the datamap from the callback code of the chroniton widget.

The more I think about this, I believe that something like a pubsub design pattern will be helpful here. htmlwidgets can provide basic pubsub infrastructure and widget authors can provide callbacks that allow users to broadcast data on certain events. In a similar vein, widget authors can also expose callbacks that allow a widget to listen to certain events and trigger a change.

By following some conventions regarding how these events are named, it would be possible for widgets to talk to each other using pubsub. Currently, this idea is still very hazy in my mind, but I wanted to put it out here so that it can trigger a broader discussion.

In the shiny case, things are easier, since Shiny.onInputChange provides a simple pubsub mechanism. I am thinking of a more general mechanism that will work with and without Shiny.

@jcheng5 I would really appreciate any thoughts you have on this, given your experience working with javascript libraries. @jjallaire @timelyportfolio @yihui Any thoughts/comments on this idea would be welcome, as I think it would be a very powerful way forward for htmlwidgets to create a componentized architecture.

@timelyportfolio
Collaborator

I agree pubsub would most likely be the best pattern. While crossfilter is nice in dc.js I think it will prove too limited for multiple htmlwidgets and difficult to teach to users. Ideally, the same event information will get passed to Shiny.onInputChange in shiny mode as our pubsub engine when not in Shiny mode, or we could allow the pubsub to notify shiny.onInputChange. I have been monitoring the work on vega closely and experimenting with various pubsub libraries. I don't have a favorite.

@abresler
abresler commented Mar 6, 2015

This looks amazing I can't wait


Alex Bresler
abresler@asbcllc.com
​www.asbcllc.com​
917-455-0239​ (cell)​
On Mar 5, 2015 8:10 PM, "Ramnath Vaidyanathan" notifications@github.com
wrote:

I have been experimenting with the ability to use widgets as components
and then allow them to communicate with each other.

Here are some examples

  1. Life Expectancy Choropleth + Line
    https://www.youtube.com/watch?v=fbmSHQrrJrI
  2. Datamaps + Chroniton https://t.co/9KzVYkbErf

The first example uses shiny, while the second does not.

In the case of Shiny, I am using Shiny.onInputChange to trigger changes
and communicate back to the server from the client on mouseover. In the
second example, I am directly manipulating the datamap from the callback
code of the chroniton widget.

The more I think about this, I believe that something like a pubsub
design pattern will be helpful here. htmlwidgets can provide basic pubsub
infrastructure and widget authors can provide callbacks that allow users to
broadcast data on certain events. In a similar vein, widget authors can
also expose callbacks that allow a widget to listen to certain events and
trigger a change.

By following some conventions regarding how these events are named, it
would be possible for widgets to talk to each other using pubsub.
Currently, this idea is still very hazy in my mind, but I wanted to put it
out here so that it can trigger a broader discussion.

In the shiny case, things are easier, since Shiny.onInputChange provides
a simple pubsub mechanism. I am thinking of a more general mechanism that
will work with and without Shiny.

@jcheng5 https://github.com/jcheng5 I would really appreciate any
thoughts you have on this, given your experience working with javascript
libraries. @jjallaire https://github.com/jjallaire @timelyportfolio
https://github.com/timelyportfolio @yihui https://github.com/yihui
Any thoughts/comments on this idea would be welcome, as I think it would be
a very powerful way forward for htmlwidgets to create a componentized
architecture.


Reply to this email directly or view it on GitHub
#86.

@ramnathv
Owner
ramnathv commented Apr 3, 2015

Here is another example using the pubsubz architecture.

http://bl.ocks.org/ramnathv/raw/51f61a43f81910b868a5/

The idea is rather simple. The morris and dimple widgets listens to an event named elid_group and updates the chart accordingly. The datamaps widget broadcasts the state being hovered on to both these widgets to effect the change. It is very similar to what Shiny.onInputChange does. The question is can we write these events in such a way that both static and shiny contexts can take advantage of it.

@happyshows

@ramnathv is there any updates on how htmlwidgets will communicate with each other in future? Also, could you share the code for the examples?

@ramnathv
Owner

@happyshows this is a hard problem to solve in general. So it will take some time before we have more clarity on this. As for code, these are all widgets under development. So once I push them to github, I will also release the examples.

@happyshows

@ramnathv thanks. In future will these code be under htmlwidgets or independent repo? I'm just trying to find a way to trace the progress.

@timelyportfolio timelyportfolio referenced this issue in hrbrmstr/taucharts Aug 18, 2015
Open

Add list of to-dos with priority #1

@timelyportfolio
Collaborator

Just circling back on this. Any thoughts on how to implement this?

@ramnathv
Owner

The more I think of it, pubsub seems to be the answer. I will write up whatever I have discovered so far so that we can push this forward.

@jcheng5
Collaborator
jcheng5 commented Aug 19, 2015

FWIW, I'm planning to look at this problem in earnest in October, probably in collaboration with @jjallaire and @hadley. I have a hard time imagining how the answer would not involve some kind of pubsub/event bus, but the semantics will need to be chosen very carefully.

@ramnathv
Owner

@jcheng5 I would love to be a part of this discussion if possible.

@jcheng5
Collaborator
jcheng5 commented Aug 19, 2015

@ramnathv Yeah, for sure.

@happyshows

@ramnathv I'm also interested in clippyr (visual help guide) , is it on github under different name?

@ramnathv
Owner

@happyshows I will put it up on github this weekend.

@happyshows

@ramnathv , is clippyr on github now?

@timelyportfolio
Collaborator

@jcheng5 & @ramnathv , it's October :) and I'm ready when you are for this discussion.

@jcheng5
Collaborator
jcheng5 commented Oct 13, 2015

Just want to let you guys know that I am actively working on this. @jjallaire, @hadley, @wch, and I were all together in person last week and discussed this at length. I'll soon have a strawman writeup and prototype to serve as a starting point for further discussion.

@ramnathv
Owner

Great! Looking forward to this @jcheng5 !

@timelyportfolio
Collaborator

very happy to hear. let me know if I can help.

@jcheng5
Collaborator
jcheng5 commented Nov 26, 2015

Sorry for the delay. I had to reboot this effort twice, I didn't realize how big the solution space was when I first started thinking about this and it took a while to get to an approach that was both practical and flexible.

I'm thinking about things in terms of a "near-term plan" that is actionable today and involves code we definitely can all agree needs to be written, but is modestly scaled in terms of ambition. Then we wait to see what kind of emergent behavior comes out of those improvements, and what opportunities we have to improve htmlwidgets to help out. Though below I also propose one possible direction that might be interesting to explore.

Near-term plan

The approach I'm advocating has two "prongs" today that address the needs of two different types of user. One is the R user who doesn't know JavaScript, and the other is the intermediate-to-expert JavaScript programmer.

For R users who don't know JavaScript:

We want to introduce a mechanism that will allow them to do things like linked brushing, linked identify, and dc.js/crossfilter-style filtering, without any effort on the users' part; just rendering three widgets that share a (new) "group" attribute value should link them together. Enabling this will take a significant amount of effort on widget authors' parts though, and we will have to provide both education and encouragement. The payoff is pretty huge if we can get this right.

That's the ultimate goal. The first step we want to take in this direction is to introduce a layer that different widgets in a page can use to speak to each other. I've built a prototype of such a layer in a tiny package I'm calling crosstalk. There's a NOTES.md document that outlines the capabilities, and the repo is here: https://github.com/rstudio/crosstalk

(BTW, for posterity, the link to the intro document is pinned to a specific commit; so if you're looking for the most up-to-date version of that file, be sure to go to the master branch first.)

I threw together a minimal d3 scatter plot package to explore these ideas without getting distracted by the complexity of most real-world widgets. The repo is at https://github.com/jcheng5/d3scatter and you can see an example of it in action with crosstalk here: http://rpubs.com/jcheng/crosstalk-demo

Crosstalk is designed not just for widget-to-widget communication, but also widget-to-Shiny and Shiny-to-widget communication as well. Try the examples on the d3scatter README. I think it'd also be very easy to extent Crosstalk to support other viz frameworks like RCloud, Bokeh.js, and Plotly.

I would love to get feedback on this approach over the next month. The high-level selection API mentioned in the NOTES.md still needs to be completed; I'm hoping to finish that next week.

For JS programmers:

The above approach is only going to be helpful for some common, standardized behaviors. It's not that helpful for totally ad-hoc interactions between widgets. For programmers proficient in both R and JavaScript, it can be challenging to get htmlwidgets to work together even with custom JavaScript code, without resorting to some ugly hacks. Primarily this is because we don't provide an easy way for htmlwidgets to have arbitrary methods on their objects; and even if we did, we don't have an easy way for regular JavaScript code to get ahold of widget objects. I have addressed the first part of that in this PR: #171

If we agree on that approach the second part of it (making it easy to get ahold of widget objects) will be straightforward. If these changes are noncontroversial they could be on the master branch before the end of next week.

Future plans

So non-JS-wielding R users will be able to do basic interactivity, and JS-savvy R users will be able to do anything they want (by writing custom JS code). I suspect there may be a fertile ground in between, where we can allow non-JS R users to do some declarative stuff in R that is way more flexible than just basic interactivity, but still keeps them from having to learn JS. I'm thinking maybe something akin to the Cocoa Target-Action pattern. This is pretty speculative though, and depends on the adoption of Crosstalk in the first place. So we'll wait to see what the reaction is to the first two prongs before we move forward with this more ambitious idea.


Feedback appreciated. If this isn't making sense to people I'm happy to vchat, or make a brief screencast or something, next week after the Thanksgiving holiday.

@jjallaire
Collaborator

@hafen and @cpsievert this is a general purpose mechanism for widget-to-widget and shiny-to-widget communication that we are working on now. We'd love to make sure that there are no major impedance problems with rbokeh and plotly as getting those packages to work with this scheme would be a huge win. Let us know your thoughts.

@jjallaire
Collaborator

@hafen and @cpsievert , The overall design is here:

https://github.com/rstudio/crosstalk/blob/master/NOTES.md

And here is a demo of what the R code currently looks like (with and without Shiny):

https://github.com/jcheng5/d3scatter/blob/master/README.md

@timelyportfolio
Collaborator

@jcheng5, @ramnathv will group be the universal argument across all htmlwidgets to express which htmlwidgets will listen on the same "channel"? I only ask because I know that in rCharts we used group/groups as an argument similar to lattice. If we plan for group to be the argument, then we will need to change our rCharts2 API. Thanks.

@jcheng5
Collaborator
jcheng5 commented Dec 1, 2015

Yeah leaflet has a similar problem with the word "group". We can change it.

@timelyportfolio
Collaborator

What do other pubsub implementations use? Would channel be an option?

@timelyportfolio
Collaborator

Another question out of ignorance, would this work with proxy mechanisms such as those in leaflet and I think soon in visNetwork? I would think yes, but thought I would bring it up just in case.

@jcheng5
Collaborator
jcheng5 commented Dec 1, 2015

It really depends on the implementation to do it correctly, but I would imagine yes.

@jcheng5
Collaborator
jcheng5 commented Dec 2, 2015

The "For JS programmers" section from #86 (comment) is now feature complete in #172.

@cpsievert
Contributor

Great stuff @jcheng5! So excited to work on this. I have some work in progress here and have some basic examples here.

Make sure you install the right branch before running examples:

devtools::install_github("ropensci/plotly@feature/events")

A few things/questions:

(1) For brush events, plotlyjs natively supports one mode: zoom. I have mentioned to the plotlyjs engineers (@etpinard, @alexcjohnson, @chriddyp, @cldougl, et. al.) that adding support for more modes (e.g., select box a la bokeh) would be a huge help, and there is intention on working on this.
(2) group has a different meaning in plot_ly(), so I used set instead (this can change, of course).
(3) Is there an elegant way to make an inline-block of htmlwidgets via htmltools?

@cpsievert
Contributor

I'm also having a bit of trouble figuring out how to pass selections from plotly to shiny without the use of an input brush. Is this what crosstalk::ClientValue is for?

@jcheng5
Collaborator
jcheng5 commented Dec 2, 2015

Yes, ClientValue is the low-level abstraction for getting crosstalk values into R. I'm also working on a SharedData abstraction that is specifically for data frames with selection. I should be able to send you some sample code tonight or tomorrow. If that doesn't make it natural for you I'd love to vchat with you this week and understand a little more about the plotly JS API.

@cpsievert
Contributor

That'd be great! I imagine it works something like this? For some reason I get NULL even after clicking on points

@jcheng5
Collaborator
jcheng5 commented Dec 5, 2015

@cpsievert It looks to me like:

  1. When building the widget payload p in the R code, you're not actually including the set value, around this area: https://github.com/ropensci/plotly/pull/312/files#diff-eb044f253f9369d7d0711127e6035fbbR100
  2. In your renderValue, I set a breakpoint on the ctgrp.var("tdb").set(keys); line, and it never seems to be hit.
  3. In your server.R, crosstalk::ClientValue$new("tdb") should be crosstalk::ClientValue$new("tdb", group = "A").

If you load your app and from the JS console you run crosstalk.group("A").var("tdb").set("foo") you should see the output update.

@cpsievert
Contributor

Thanks @jcheng5! I fixed (1) and (3), and I think (2) was happening because I wasn't extracting the key properly for that example. It seems like it could be useful to expose more than just the key, so in ropensci/plotly@d0b6118, I'm just trying to set/get all the point data, which seems to work client-side, but I'm getting Uncaught TypeError: Converting circular structure to JSON from shiny.min.js

Any ideas?

PS: I changed the naming a bit, but I revised the gist to reflect that

@cpsievert
Contributor

Actually, I think this is a plotlyjs issue. If I try to call JSON.stringify(data.points) from here it gives me the same error. Any ideas @etpinard / @alexcjohnson / @chriddyp / @jackparmer?

@jackparmer

++ @mdtusz ^^^

@etpinard
etpinard commented Dec 7, 2015

@cpsievert data.points[i].xaxis and data.points[i].yaxis are indeed circular structures.

They refer to the points' corresponding cartesian axis objects which you shouldn't need for your use case.

I'd recommend doing something like:

data.points.forEach(function(pt) {
  delete pt.xaxis;
  delete pt.yaxis;
});
JSON.stringify(data);
@alexcjohnson

@cpsievert @etpinard even better, pull out just the pieces you really want from data.points. Each point has the COMPLETE trace and fullData (after supplyDefaults) trace, the latter of which contains references to code in _module... wayyyy more than you want. I'd prefer:

JSON.stringify(data.points.map(function(pt) {
    return {
        curveNumber: pt.curveNumber,
        pointNumber: pt.pointNumber,
        x: pt.x,
        y: pt.y,
        key: pt.data.myKey[pt.pointNumber] // if you need something else as the key
    };
});
@cpsievert
Contributor

Thanks @etpinard @alexcjohnson. I agree that we should send as little data as possible, so I bootstrapped on @alexcjohnson suggestion in ropensci/plotly@d180812

Here is a demo of handling plotly click events in shiny via crosstalk (source).

@jcheng5
Collaborator
jcheng5 commented Dec 8, 2015

Nice work @cpsievert. The demo app code looks really clean.

You could simplify further by getting rid of the rv reactive expression; you can declare cv as a variable directly inside the server function, and use cv$get() from any reactive expression, observer, or output you wish. In fact, even if you decide to keep the rv reactive for future purposes, you should still hoist the cv <- ClientValue::new(...) declaration at the top level of the server function, and just put cv$get() inside the body of rv.

@jonathancallahan

Just wanted to say that I'm very excited to see this work progress. I see mention of widget-widget and widget-shiny interactions but I didn't see any mention of "widget-R".

Will this work allow us to create widgets that interact with the RStudio environment/console?

I'm imagining an interactive widget that allows people to visually select some subset from a dataframe and that selection or that subset will somehow be communicated back to the RStudio environment. Alternatively, a widget be could used as a point-and-click interface to create an R expression that gets loaded into the console.

We are already creating a package that harnesses the dygraph and leaflet widgets to basically customize RStudio into a statistical analysis tool for non-programmer, Forest Service Personnel. It sure would be awesome if there were a mechanism by which scientists could click on a monitoring site in the leaflet widget and have their selection immediately available in the current R environment.

@jcheng5
Collaborator
jcheng5 commented Dec 12, 2015

@jonathancallahan Yes, but you don't even need htmlwidgets for that--anything you can do in Shiny can be harnessed by the current R environment.

Try this (requires devtools::install_github("rstudio/shinygadgets"); and looks best in RStudio, although it'll work with any R):

library(shiny)
library(shinygadgets)
library(leaflet)

ui <- fillPage(
  leafletOutput("map", height = "100%")
)

server <- function(input, output, session) {
  output$map <- renderLeaflet({
    leaflet(quakes) %>% addTiles() %>% addCircleMarkers(layerId = as.character(1:nrow(quakes)))
  })

  observeEvent(input$map_marker_click, {
    stopApp(quakes[as.numeric(input$map_marker_click$id),])
  })
}

runGadget(ui, server)
@hafen
Contributor
hafen commented Dec 17, 2015

Sorry I'm late to the game - been doing an insane amount of travel and catch-up when not traveling. This looks awesome and I'm really excited to dig into it. I'll be more involved at the beginning of the year.

@timelyportfolio
Collaborator

@ramnathv, any progress on a new scaffold template? I'm almost thinking of working on this as my htmlwidget of the week. Let me know if I can help.

@timelyportfolio
Collaborator

Or, I could convert one with a writeup to publicize the new direction for the year. Any particular htmlwidgets that provide an easy use case? Would knob be appropriate since it was a test for the first implementation?

@ramnathv
Owner

@timelyportfolio I already have a local branch with the new scaffold template. I am on vacation currently, but will push it out over the weekend. I will be including an example as well.

@timelyportfolio
Collaborator

Now that my widget per week commitment is over, I can play with crosstalk :) Here is d3scatter with parcoords http://bl.ocks.org/timelyportfolio/4c5718a7efe5c0abd363. Currently, I only have send not receive for parcoords. Thanks so much @jcheng5 for spearheading this.

@happyshows

@timelyportfolio great achievement for 2015 :D Could you setup a voting app with multi choices for each widget and improve the highly rated ones later on?

@timelyportfolio
Collaborator

@happyshows, thanks for all your usage/feedback on htmlwidgets. I had hoped to get a feel for popularity by Github stars, but unfortunately getting stars on Github for R projects seems nearly impossible. Unless otherwise motivated by feedback, ideas, use cases, I'll probably just pick my favorites.

@hafen
Contributor
hafen commented Jan 5, 2016

@timelyportfolio congrats on getting through the year of htmlwidgets! I'd love to get all your missing widgets onto the htmlwidgets gallery. If we made the site more prominent and added a gallery, etc., then maybe it would become a place for things like github stars to become more effective.

@happyshows

@timelyportfolio Maybe put a survey monkey link on your site to collect preference & feedback data instead? just my 2 cents.

@ThomasSiegmund

Hi all,

I'm very happy about crosstalk. IMHO this is the next big step for the Shiny and htmlwidgets ecosystem.

I have now added support for crosstalk in D3TableFilter. There is a demo app combining a d3scatter and a d3tf widget. Selecting dots in the plot highlights corresponding rows in the table and vice versa.

I guess it will take a while and a few good apps to figure out all the semantics. The table may allow to filter out rows - what should happen to the dots in the graph? Should we have an additional crosstalk variable ("hidden") ?

Thanks a lot @jcheng5 !

@timelyportfolio timelyportfolio referenced this issue in timelyportfolio/explodingboxplotR Jan 11, 2016
Closed

Widget must have a renderValue function #1

@timelyportfolio
Collaborator

@jcheng5, @ramnathv any updates on this?

@ramnathv
Owner
ramnathv commented Mar 8, 2016

@timelyportfolio mobservable has changed their API significantly. I am reworking things using the new version and will put it out here. There are still several outstanding issues in terms of the order in which observables get evaluated. Based on conversations with @jcheng5, it is bound to be much harder doing everything client side. But I think there is scope for an effective solution.

@timelyportfolio
Collaborator

@ramnathv I wondered if the change to mobx would throw a kink into the solution, and I guess the answer is yes. As always, I am very interested and happy to help in any way possible.

@jcheng5
Collaborator
jcheng5 commented Mar 9, 2016

I should probably write up a blog post or something about the current state of things--I've had the same conversation privately with a number of people. Basically there's several different directions things are growing, each of which is ideal for a particular kind of widget user (depending on how skilled they are in JS, how much time they want to spend writing custom code, and how much R runtime support [Shiny] is needed).

I had told @cpsievert and @hafen (and possibly @ramnathv) I was having second thoughts about the crosstalk approach, but subsequent conversations with @jjallaire have convinced me that it's an important piece of the story. I'm tentatively planning on investing further in crosstalk over the next few months, with an eye towards a CRAN release (or merging into htmlwidgets master?) and getting several more htmlwidget packages working well with it.

@timelyportfolio
Collaborator

@jcheng5 + @ramnathv thanks so much for the updates. I'm just a little excited :) and anxious to test and experiment. I stopped the crosstalk earlier this year to make sure I wasn't headed in the wrong direction.

@jcheng5
Collaborator
jcheng5 commented Mar 11, 2016

@ramnathv "The order in which observables get evaluated" sounds bad--I believe it shouldn't be a problem if you follow some simple (well, simple-ish) rules. This was one of the main topics of my 3 hour reactivity tutorial at ShinyDevCon in January. We should talk about this if this isn't something that mobservable considers a solved problem.

@ramnathv
Owner

@jcheng5 the problem is not with how mobservable deals with things. it is more about syntactic sugar that i was trying to add that automatically creates observables underlying elements in the dom. so the order issue arises due to the dom, not because of the underlying library itself. I will create a set of simple examples to illustrate the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment