Integrate various plotly packages

[davidanthoff]

I just looked around a bit for a plotly package, and after a short google, I came across at least four:

https://github.com/plotly/Plotly.jl
https://github.com/joshday/PlotlyLocal.jl
https://github.com/EricForgy/PlotlyJS.jl
https://github.com/spencerlyon2/Plotlyjs.jl

The last three seem to be based on the recently released JS package, i.e. they take a local approach. To me they all seem to try to do the same thing (maybe wrong?).

Maybe there could be some coordination, and an attempt to have one high quality plotly package at the end that provides both the local JS type usage model and the old web based approach, and ideally all of that would be in this package here, which seems the most official one?

[tbreloff]

Plotly is mostly supported in Plots.jl as well. See: JuliaPlots/Plots.jl#77. Obviously my hope is that Plots is where the coordination happens, as certain pieces can be generalized to other web-based backends.

[Evizero]

IMHO I'd prefer the online and the local version to be separate

[EricForgy]

Hi guys,

I just pushed some improvements to https://github.com/EricForgy/PlotlyJS.jl (including a simple README).

I agree with @Evizero that online and local versions can be separate packages, but one thing I think would be useful for any Plotly-related package is just a high-level sugar layer that creates Dicts. These Dicts could be manipulated as desired and finally passed through JSON.jl to convert them to something digestible by plotly.js.

I need to catch up with what @tbreloff and @spencerlyon2 are doing. @tbreloff, I imagine to make the progress you've made, you probably had to write some of that "sugar" already, right? It might be good to strip that out to a separate package that could be used by everyone.

For my purposes, I want to avoid node.js and just have Julia talk directly to a browser.

Cheers

[bpostlethwaite]

We are definitely interested in working the community on a solution (see related issue #19). I am a little busy until after the holidays but am looking forward listening to ideas and helping out where I can.

Happy holidays!

[tbreloff]

you probably had to write some of that "sugar" already, right? It might be good to strip that out to a separate package that could be used by everyone

Based on this comment, I'll assume that you haven't actually looked at Plots.jl yet. Plots has excessive amounts of generalized "sugar". That's what it is. It's generalized in that the same api/code is used to produce simple line plots in the repl (UnicodePlots) and complex 3D animations. It's not "generalized Plotly"... it's "generalized Plotting".

There is also an ecosystem that's starting to sprout up which uses the Plots framework... see https://github.com/tbreloff/MLPlots.jl. I expect other similar projects to follow (FinancePlots.jl, BioPlots.jl, NeuroPlots.jl, etc)

I am obviously biased in my view, but I think any community collaboration that doesn't include Plots as the interface will be missing out.

Now, if I can assume for a moment that Plots is the primary interface, then where do the other packages fit in:

PlotlyLocal.jl

This is @joshday's and I think he's said that it was just a prototype and that he doesn't intend on pushing development forward (correct me if I'm wrong)

PlotlyJS

@EricForgy There isn't much code in your repo, and it's primarily geared towards serving a websocket listener. I understand your goal, but I don't think it has much to do with Plotly. If your goal is to produce the layout and data dictionaries that you can pass to Plotly, then I would recommend using Plots, and instead of displaying the Plots.Plot directly, you could call the Plots internal methods which do this, and pass them through your websockets.

julia> using Plots; plotly()
Plots.PlotlyPackage()

julia> p = plot(rand(10), c=:red, m=(20,:green));

julia> Plots.plotly_layout(p.plotargs)
Dict{Any,Any} with 9 entries:
  :yaxis         => Dict{Symbol,Any}(:linecolor=>"rgba(0, 0, 0, 1.000)",:type=>"-",:titlefont=>Dict{Symbol,Any}(:size=>15,:color=>"rgba(0, 0, 0, 1.000)",:family=>"Helvetica"),:tickcolor=>"rgba(0, 0, 0, 1.000)",:title=>"",:showgrid=>true…
  :plot_bgcolor  => "rgba(255, 255, 255, 1.000)"
  :showlegend    => true
  :legend        => Dict{Symbol,Any}(:bgcolor=>"rgba(255, 255, 255, 1.000)",:font=>Dict{Symbol,Any}(:size=>11,:color=>"rgb(0, 0, 0)",:family=>"Helvetica"),:bordercolor=>"rgba(0, 0, 0, 1.000)")
  :titlefont     => Dict{Symbol,Any}(:size=>15,:color=>"rgba(0, 0, 0, 1.000)",:family=>"Helvetica")
  :paper_bgcolor => "rgba(255, 255, 255, 1.000)"
  :xaxis         => Dict{Symbol,Any}(:linecolor=>"rgba(0, 0, 0, 1.000)",:type=>"-",:titlefont=>Dict{Symbol,Any}(:size=>15,:color=>"rgba(0, 0, 0, 1.000)",:family=>"Helvetica"),:tickcolor=>"rgba(0, 0, 0, 1.000)",:title=>"",:showgrid=>true…
  :title         => ""
  :margin        => Dict(:r=>8,:l=>35,:b=>30,:t=>20)

julia> Plots.plotly_series(p.seriesargs[1])
Dict{Any,Any} with 7 entries:
  :y      => [0.32097734976939707,0.687242060745739,0.39992385307159384,0.21123519739765628,0.8344248977838238,0.32680962841391215,0.2688450717696713,0.7247504231458914,0.0572384724490953,0.8498480241370621]
  :name   => "y1"
  :type   => "scatter"
  :line   => Dict{Symbol,Any}(:width=>1,:dash=>"solid",:color=>"rgba(255, 0, 0, 1.000)",:shape=>"linear")
  :x      => [1,2,3,4,5,6,7,8,9,10]
  :mode   => "lines+markers"
  :marker => Dict{Symbol,Any}(:symbol=>"ellipse",:opacity=>nothing,:line=>Dict{Symbol,Any}(:width=>1,:color=>"rgba(0, 0, 0, 1.000)"),:size=>40,:color=>"rgba(0, 128, 0, 1.000)")

I think you should consider renaming your repo and generalizing your approach to do more than just Plotly. I also think you should review Blink and see how much overlap there is.

Plotlyjs

@spencerlyon2 and I are working on integrating his approach into the Plots flow. For now we are focused on Plotly, but I hope to eventually generalize certain parts to other web-based plotting backends. He is leveraging Blink to fill in an important need... to have standalone "desktop gui" windows that are powered by web technologies. This approach also allows workflows in Atom and IJulia/Jupyter which are typically frustrating.

Plotly.jl

This falls into a separate category. Based on my limited understanding of Plotly's cloud architecture, I think I would recommend that this package stays independent, but that it uses the Plots interface as I described above... building the dictionaries through Plots, then passing those off to a (smaller) package which handles the authentication, etc needed for the cloud.

Summary

I think:

• Plots.jl should be the core user interface.
• Plotlyjs.jl should be the primary method of displaying through Plots.
• PlotlyJS.jl should be renamed and re-focused on it's true purpose... passing javascript updates through websockets.
• Plotly.jl should be kept mostly as-is, but depend on Plots to create the plot JSON.

Obviously these aren't my packages, and you can do whatever you want with them, but I think this path forward is the best for the community.

[davidanthoff]

My preference would be to have one plotly package that is a closely aligned with the original plotly API as possible and aims at fully representing that. Ideally it would cover the local/JS and cloud approach, so that I could just change one line of code depending on whether I want to show the plot locally or push it to the plotly cloud.

I think the Plots.jl initiative is great, but I wouldn't want the only julia interface to julia be a package that largely tries to unify plotting APIs. Wouldn't the right layering be that there is a Plotly.jl package that is a faithful to the plotly API as possible, and then Plots.jl could utilize that and provide a different API on top? I.e. that Plotly.jl is one more backend for Plots.jl?

[sglyon]

Thanks for starting this discussion. I'll share my thoughts.

Personal update: My work on Plotlyjs.jl is mostly exploring how we can have a dedicated display system for plotly graphics and tightly integrated interop between Julia and javascript. Leveraging Blink.jl (electron) gives us both things. Blink could be a bit more polished and tested, but once you get it working it does a great job. I've been able to expose all the exported functions from the plotly.js api and am working on some of the un-exported ones also (mostly as needed).

The rest of Plotlyjs.jl is my quick and dirty attempt to provide some of the sugar @EricForgy talked about above (mostly so I have the ability to quickly spin up plots while testing the js interaction + Blink display). The relevant methods are here. The resultant API is actually quite clean and can been seen in these examples (Note that in those examples I show all the different entry points into the API, not necessarily the most concise in each case). The cool thing about the methods is that they allow you to set "deep" plot attributes using keyword arguments containing underscores. For example, I can do this:

    trace1 = scatter(;x=1:5, y=[1, 6, 3, 6, 1],
                      mode="markers+text", name="Team A",
                      textposition="top center",
                      text=["A-1", "A-2", "A-3", "A-4", "A-5"],
                      marker_size=12, textfont_family="Raleway, sans-serif")

instead of

trace1 = Dict("type" => "scatter", "x" => 1:5, "y" => [1, 6, 3, 6, 1]
              "mode" => "markers+text", "name" => "Team A", 
              "textposition" => "top center",
              "text"=>["A-1", "A-2", "A-3", "A-4", "A-5"],
              "marker"=>Dict("size"=>12), 
              "textfont"=>Dict("family"="Raleway, sans-serif")

Notice how much lighter the syntax is, especially for setting marker.size and textfont.family.

I also have a super lightweight way to construct subplots via the familar hcat, vcat, hvcat family of functions. By this I mean to create a subplot with 2 rows and 1 column I would first build two plots p1 and p2 and then do p = [p1, p2] (vcat). If instead I wanted a 1x2 layout I'd just do [p1 p2] (hcat). Likewise to do a 2x2 grid I would do [p1 p2; p3 p4] (hvcat). It handles sizing and moving individual plot titles to annotations, preserving tick settings, axis labels and anything else that may have been customized on the axis of the original plots.

---

Thoughts about Future

With that update in mind, I should say that I don't anticipate spending much (any?) more time fleshing out an API for constructing plots. For moderately simple plots I'd be fine to use what I have in Plotlyjs.jl, but for more complicated or structured plots I will almost certainly turn to Plots.jl. @tbreloff has done an excellent job thinking through a usable/flexible API for plotting in Julia and I'm excited about the opportunity to tap into that system. For me, generating the figures with Plots.jl and them displaying them with Blink/Electron seems to be the way forward.

[EricForgy]

Thanks for the update @spencerlyon2 and thanks for the thoughts @tbreloff.

I think the way forward is clear and pretty much what Tom suggested. I like Spencer's Plotlyjs.jl and would suggest renaming it to PlotlyJS.jl to be consistent with ThreeJS.jl. This naming convention can set a standard for future Julia interfaces to JS APIs. Then PlotlyJS is the high-level sugar I was looking for. I agree, the PlotlyJS syntax is nice and cleaner than constructing Dicts manually. Then Plots.jl can provide "sugar on sugar" to interface with PlotlyJS.jl and kind of bring everything together (if I understand Plots.jl).

Finally, what to do with Plotly.jl? My thought there is that it can be an interface for Plots.jl to get other packages onto the Plotly platform, e.g. generating an interactive web-based Gadfly plot or something, but not sure :)

At some point, I will rename my repo from PlotlyJS.jl to something else (so no need to worry about name clashes), but am not in a huge rush unless keeping the name causes problems for others. As Tom mentioned, mine is really just a WebSocket listener. I'd like to mimic a lot of what Blink can do (including building a browser based Julia IDE), but do it in a standard browser instead of Atom, but that is a subject for another place.

[sglyon]

Hey @EricForgy it's outstanding to have your approval on this. I think that the WebSocket listener will be quite useful and a contribution in and of itself (lower the bar for non-web people like myself to leverage solid packages). I'm excited about Blink because electron is a pre-packaged solution that handles most of that nitty gritty stuff for me.

I'm happy to rename to PlotlyJS.jl.

Regarding what to do with this package (Plotly.jl)? I would vote that it's main purpose remain interfacing with the plotly REST api. I think various other packages (PlotlyJS, Plots, Gadfly, etc.) can send prepared JSON to Plotly.jl, where the authentication and other REST api specific things can be handled. Whether the interface between Plotly.jl and other packages lives here or elsewhere is another question that needs to be answered.

This structure also seems to be in line with #19. If this package doesn't focus on providing a Julian plotting API, but rather communication to/from the plotly servers, then the core plotly team might be the best group to maintain the package? If more Julian expertise is required/desired I'm sure the community (including myself) would be willing to pitch in.

[EricForgy]

PS: I am preparing a PR to rename Plotlyjs to PlotlyJS so that might save you some time. I'll try to get it done quickly.

Edit: Submitted.

Edit^2: I renamed mine to JuliaJS for the time being to get out of the way :)

[malmaud]

@spencerlyon2, myself, and @bpostlethwaite chatted today about this issue. Here's what we're thinking:

• This package will support both remote and local mode and will aim to be the canonical Julia package for Plotly
• It will generally resemble the Python Plotly package in terms of its API
• It will depend on https://github.com/spencerlyon2/Plotlyjs.jl for most of its functionality (the API for specifying graphs and rendering them to Electron windows)
• But will additionally add support for using the REST API to interact with the plot.ly service (local-only mode will still be fully supported)

How's that sound to everyone?

[davidanthoff]

I think that sounds fantastic! The cleanest, easiest to understand arrangement in my view.

[malmaud]

Closed by #21