RFC: unify plotting packages under a common organization

[waldyrious]

I originally suggested this in sunetos/TextPlots.jl#15, where it was briefly discussed, but since the subject was kind of off-topic there, and the PR was (rightfully) closed, I thought it would be better to re-launch the discussion elsewhere. I'm posting it here since it's by far the most popular (by github stars at least) plotting package for Julia.

The suggestion is for the owners of Julia's plotting packages to join under an organization, which would have several benefits, which I point out below. For reference, the current plotting packages are:

Text/terminal-based plotting:

• @sunetos' TextPlots.jl
• @johnmyleswhite's ASCIIPlots.jl
• @mbauman's Sparklines.jl
• @Evizero's UnicodePlots.jl
• @ninjin's Hinton.jl (also outputs vector graphics)
• @meggart's ImageTerm.jl

Graphical plotting (possibly incomplete list):

• @dcjones' Gadfly.jl
• @stevengj's PyPlot.jl
• @nolta's Winston.jl
• @mbaz's Gaston.jl
• @Mononofu's julia-plot
• @SimonDanisch's GLPlot.jl
• @tbreloff's Qwt.jl
• @bokeh's Bokeh.jl
• @plotly's Plotly.jl
• @jverzani's GoogleCharts.jl
• @johnmyleswhite's Vega.jl
• @wildart's PLplot.jl

Additionally, there are some graph-specific packages which I suppose would fit into a plotting organization, but since JuliaGraphs exists, they should probably unite there instead:

• @Keno's GraphViz.jl
• @sisl's TikzGraphs.jl
• @IainNZ's GraphLayout.jl

The advantages for an organization are:

1. It would make it less harmful if the de facto maintainer of a package became absent or semi-absent for a long period of time, since other members of the organization could at least merge simple fixes (compatibility changes, readme edits, documentation, etc.), and new people who start contributing a lot to a given package could be added to the organization to help maintain the package.
2. It would provide a curated list of plotting packages, making it much easier to find the various options available (it took me quite a while to collect the list above, sifting through long julia-users threads and other online resources). For example, http://julialang.org/downloads/plotting.html only mentions Gadfly and PyPlot.
3. As @tbreloff mentioned in the previous discussion, such an organization would be the ideal place to maintain a "document/tutorial that describes the pros/cons of the packages, compares usage for common workflows, dependencies, and features", further adding that he'd be interested in contributing (as would I).
4. More of a long shot, but nothing is lost for trying: maybe that organization could facilitate convergence efforts to unify plotting options in Julia. At the very least, there would be a central place for discussions to happen.

As I see it, we have nothing to lose, and everything to gain: at worst we will have a situation just like right now, but with the benefit of having all plotting platforms in the same place. That's already a net positive, without the need for any further action. And at best, we have the chance to really improve the state of plotting in Julia, which arguably is a quite common need, therefore hopefully commanding enough interest and energy to make things go forward, concerning items 1, 3 and even 4 from the list above.

Looking forward to hear from the authors of the packages listed above (and any others interested in this topic).

[Evizero]

I'm rather new to the community, so I'm lacking the experience to really help this discussion. But from where I am standing, I do think you make a lot of good points.

I would be happy to contribute where I can, although I'm actually a machine learning guy rather than a computer graphics expert

[johnmyleswhite]

I think @IainNZ's previous comment on this topic was dead on: previous organizational efforts have already demonstrated that having an organization doesn't have any substantial effect on code maintenance because the real limiting factor is the number of software engineering hours per week we receive from competent volunteers. Since a large fraction of the people who've done the engineering work on graphics in the past are busy these days, I think you should assume that having a organization will not lose us anything, but will also gain us nothing. In short: the existence of an organization is neither necessary nor sufficient to create a labor supply.

If you'd like to help make graphics better, the best way to do that is spend your time writing graphics code. Become the labor supply you want to see in the world.

[tbreloff]

How does one properly form a group like this? I say we just do it, and start working on a comprehensive doc as I described (compare and contrast the various plotting options).

I can start work on this... The most helpful thing would be if people could describe their common workflows and how they would prefer to interact with a plotting package (DataFrames vs Arrays, lines vs histograms, etc), how often then need certain features/styles, etc, whether they prefer minimalist, desktop gui, web, etc.

I know how I like to work, and that's why I built Qwt.jl. Showing some other sample workflows and how you would accomplish it in all the packages would be super useful for all, and it might help guide an eventual unifying package at some point.

On Aug 15, 2015, at 9:56 AM, John Myles White notifications@github.com wrote:

I think @IainNZ's previous comment on this topic was dead on: previous organizational efforts have already demonstrated that having an organization doesn't have any substantial effect on code maintenance because the real limiting factor is the number of software engineering hours per week we receive from competent volunteers. Since a large fraction of the people who've done the engineering work on graphics in the past are busy these days, I think you should assume that having a organization will not lose us anything, but will also gain us nothing. In short: the existence of an organization is neither necessary nor sufficient to create a labor supply.

If you'd like to help make graphics better, the best way to do is spend your time writing graphics code. Become the labor supply you want to see in the world.

—
Reply to this email directly or view it on GitHub.

[dcjones]

A document comparing different plotting options for Julia would be very useful. We don't really need an organization to do that though, just a repository and some people willing to work on it.

[timholy]

I actually think that packages owned by individuals sometimes get more maintenance. I'm not sure why, but I think it's because at least someone feels responsible that way.

[waldyrious]

To address @johnmyleswhite's (and @IainNZ's) points, the intention was never to generate new energy/labor (or in any way ask the package managers to do more than they are currently doing!), but merely to set up the structure that would allow e.g. convergence efforts once people are available to do so. In the meantime, it would provide a central location to find all plotting packages (definitely not "nothing" to gain), it would be the ideal host for the document @tbreloff talks about (which I reiterate my willingness to help work on), and especially, it would allow existing "labor supply" (unmerged PRs, especially simple ones) to be better channeled.

In fact, let me expand on this latter point for a bit: It may not be evident for you as someone who has commit access to many of the ecosystem's main packages, and commands the prompt attention of community members, but from an outsider's point of view, having extremely simple patches linger for weeks or even not merged at all definitely doesn't encourage more substantial contributions. Add to this the fact that by the time a PR is merged the person may not have as much time available as they did when they sent the first PR, or may have unloaded the mental model of the package (which takes significant effort to acquire when dealing with someone else's code), or the model became unsynchronized with recent upstream changes to Julia, etc., and it may become clearer why I feel this aspect is so important to address. I'm positive that an organization could provide a more responsive ramp-up for new contributors (i.e. the labor supply you're talking about), and I say this from experience: my (modest) PRs to julia proper have had much prompter responses from various people, while contributions to single-owner, less popular packages typically take longer to have responses, for obvious (and understandable) reasons. @timholy's observation to the contrary probably only applies to a limited subset of packages whose owners are both very active in julia development and take care to address PRs/issues early, which isn't the general case (but please correct me if I'm wrong).

For all the above, I ask you to reconsider the potential benefits of an organization. It may not mean much to powerhouse contributors, but it most definitely doesn't have a null effect.

[Evizero]

How about we meet half way. I kind of see @timholy 's point I have to admit.

Here's what I think would be a good idea: Create an organisation and put the documentation there that we keep talking about. Also - and I might be biased here - it would be nice if we could reduce some duplicated efforts by having some kind of base package where we put calculations such as choosing a good plotting range and tick marks; things where there are best practices and that all plotting packages need.

[timholy]

it would be nice if we could reduce some duplicated efforts by having some kind of base package where we put calculations such as choosing a good plotting range and tick marks; things where there are best practices and that all plotting packages need.

As someone who is spending about 70 hours of work extracting & extending the file I/O code from Images (into FileIO) and revamping Color (the need for which again arose from extensions in Images, and their usage in OpenGL packages), I can assure you that such refactorings are a nontrivial amount of work.

Therefore, that's a concrete way that people who want such things to happen can help out! The appropriate procedure:

• on your own machine, split out the functionality you want into a stand-alone private package
• rebase the original code to make use of your new package, get all tests working
• put your package up on GitHub (nice to put in the README some acknowledgement that this functionality was split off from package XYZ)
• submit a PR to the original package

[timholy]

My suggestion is that someone should write the document or start helping with code refactorings first, thereby demonstrating the need/benefit for this kind of thing. You can host your results under your own GitHub site initially. Otherwise, you're basically just asking authors to do a lot of useless moving things around and giving up "ownership" of things they've put a lot of work into.

[lobingera]

I consider myself at some intermediate level doing visualisations (still not an expert) and agree the state of plotting with/in julia is not at the same level as other features. I'll writer later today a longer comment, but as john has written above: Work can never be replaced by organization. But organization can simplify work by giving some structure.

[Evizero]

I think we should move away from the idea of moving the packages to an organization. I think the giving up ownership argument has a real case. Especially for big and awesome packages such as Gadfly. That being said, some sense of community and coordination would probably be beneficial.

For example (again personally motivated, but still): I very recently started with my plotting library. Aside from this discussion there is no indication to me that there is some sort of Julia-Plot-community. Would be great to have a common place to discuss things and ask questions that I am sure came up before.

As a newcomer and thus still an outsider I can also say that aside from PyPlot and Gadfly I had no idea other approaches exist at first. What I mean is, that unless you really look for specific information you won't stumble across it. That may sound unproblematic at first, but it closes doors for the spoiled that don't want to make that effort to educate themselves. It's a Julia entry-barrier.

For example, I am a machine learning PhD student who is in the process of moving over to Julia. However, I have a hard time convincing colleagues to jump ship on python/lua and join me on my Julia endeavours, when basic things like plotting need to be researched. Granted, plotting is not the main problem in my quest, but I think you get my point. Things like plotting should be obvious to the oblivious

[timholy]

Agreed that we need better coordination of information. http://julialang.org/downloads/plotting.html could probably use some love. Or maybe someone wants to start a Wiki?

[lobingera]

Is this the correct place?
https://github.com/JuliaLang/julialang.github.com/tree/master/downloads

Looks like we just need PRs for that.

[StefanKarpinski]

While orgs don't magically create manpower, they do seem to push people towards convergence of interface design, which can be a very beneficial thing. I'm not sure if that's a realistic or reasonable goal in the world of plotting, however. Can we separate the lower level functionality of something like Gadfly from the "grammar of graphics" style API it has? Or is that just Compose at that point?

[timholy]

https://github.com/JuliaLang/julialang.github.com/tree/master/downloads

I think that's right, the contents of the .md file seems to match.

[tbreloff]

I think the idea of a common plotting interface would be hugely beneficial, and I would be willing to put in time to make that happen. As I've said, I think the right first step is to create a document that has examples of implementing various workflows in each plotting package, perhaps in a public wiki. I know my own workflow, but I need examples of other people's workflows for the wiki to be useful.

Summary: can people start posting some example plots they find important with the underlying data, and I could start building the wiki?

Then we would have a point of discussion for what a common interface should look like. An organization could be created at any point... Doesn't need to be the first step.

On Aug 17, 2015, at 8:19 AM, Stefan Karpinski notifications@github.com wrote:

While orgs don't magically create manpower, they do seem to push people towards convergence of interface design, which be be a very beneficial thing. I'm not sure if that's a realistic or reasonable goal in the world of plotting, however. Can we separate the lower level functionality of something like Gadfly from the "grammar of graphics" style API it has? Or is that just Compose at that point?

—
Reply to this email directly or view it on GitHub.

[SimonDanisch]

Yes, please let us collect use cases. I would find that very beneficial as well.
I already opened an issue about that:
JuliaGeometry/meta#8

[tbreloff]

Just to get things started... here's a visualization that I do which uses Qwt.jl (the gif was also made with Qwt.jl):

http://giphy.com/gifs/l41lFqlq6DpOJGDyE

You can see the code that went into this here: https://github.com/tbreloff/OnlineAI.jl/blob/master/src/liquid/visualize.jl

Please... anyone that has visualizations/plotting workflows for which they chose a specific package... please post something similar. It will really help to design/organize/fill the wiki.

[lobingera]

I'm currently working on making interaction (based on Gtk/Cairo) happening, so zoom+pan and rubberband selection for 2D plotting. As a screen shot this will not look very impressive ...
@SimonDanisch What actually are looking for in the use case list?

[SimonDanisch]

All kinds of plots. Best would be to not only have classic examples like from the matlab interface, but some use cases that don't exist but people phantasize about.
Your interaction use cases sound like a good one! GLVisualize should be well fitted for interactive visualizations, but I'm just now trying it out with real world use cases.

I'm in this weird situation of trying to write a plotting library without actually using it.
Its even worse, as GLVisualize should be a backend for many different high-level plotting interfaces, so I want to get an overview of the problem space.
I'm doing my best to find the most natural abstractions, but I'm a little scared that they turn out to be impractical for real world use cases.

[timholy]

@lobingera, you might want to see https://github.com/timholy/GtkUtilities.jl. After I finish my current tasks, I was going to snip out the pan/zoom code from ImageView. But I'd be thrilled if you did it instead!

[lobingera]

In overall there seems to be an agreement that an organization does not solve the main problem: manpower, but something for tighter communication would be nice. My proposal: attach to JuliaGeometry, as some plotting and plotting infrastructure is happenning there anyway. Open question: Should plotting tools packages themselves move into the JuliaGeometry organisation?

Looking at plotting as a topic, some sort of grouping could be done.

1. julia interfaces to independent plot SW (like Gaston for gnuplot, i guess)
2. julia interfacing plotting libraries (like PyPlot, matplotlib)
   2a) cross-language stuff
   2b) ccall library connection
   2c) interfacing HTML5/js libraries
   2d) contact plotting (web-)services
3. julia interfacing plotting components
4. full julia solutions (maybe outsourced rending like Cairo)
5. anything i forgot

It also looks like an agreement to update/revisit the julia webpages to give everyone an overview of stable and developing plotting solutions -> PRs on julialang.github.com

@tbreloff: Search for plotting discussions on julia-users. I remember at least twice a year a 'how could a plotting API look like' discussion.

@waldyrious: I also had missing communication on progress of PRs (i think my quiver plot example on Winston is still pending), but githup enables me a workstyle that keeps me independent of this. Asking for updates can be seen a polite reminder. But don't expect wonders, in some cases the maintainers do not agree with you.

@StefanKarpinski: I spend some time on the weekend looking into Gadfly and Compose because i wanted to propose some code to paint the bitmap/image into cairo based backends (but got lost in color conversions, which is a nightmare of it's own). I thing Gadfly is clearly build on a Compose backend, but Compose could deal as a layout engine, that abstracts the current backends. Afaics.

@SimonDanisch: I'll find some time for brainstorming fill you discussion in juliageo-meta.

[lobingera]

@timholy, yes i'm aware (and actually at one point of time proposed to spin off 'higher' gtk functionality into a package) of GtkUtilities.jl.

I wanted to have good solutions for

• pan
• zoom on mouse wheel
• selection by rubberband
• selection by click (multi-click by shift-click or similar, modifyer key)
• tooltip like data info
• draft drawing with a limited reaction time to have interaction and precise drawing in the background for plotting cases with many (>1e6 elements).
• add elements by drawing with mouse (adding a line/multiline, markers, text boxes etc.)
  before going into publishing or discussion

[lobingera]

... and i only have solutions for the first three...

[timholy]

@lobingera, I guess my point is that if you already have partial solutions, better to unify on a single package and put it up there. GtkUtilities exists, but I'm happy to abandon it if you already have something better.

However: over the next week or two I'm going to push my new branch of ProfileView that uses Gtk. Currently it uses GtkUtilities (which I'm happy to change), but if nothing else better is publicly available I will be adding more features to GtkUtilities. Hence, waiting a long time to even post your work will result in fragmentation.

Hence, if you have something better available I urge you to post it somewhere. I am happy to contribute to your package.

[waldyrious]

There are certainly lots of paths to move forward with this, and I'm thrilled to see this thread has sparked a lively conversation. However, I believe we should, as first steps, make the small but consequential changes that would provide a stable, permanent platform for the remaining discussions to progress.

At least two fundamental changes are needed to ensure the discussion from this thread doesn't die off (or become one of those huge, impractical catch-all threads), in my view:

1. We need a central location for users and potential developers to find all existing plotting packages. An organization would seem like a natural choice for this, but if people don't want this (although to be honest I haven't seen any arguments actually against it, just assertions that it wouldn't bring any benefit, which I disagree with as pointed out earlier), then how about using http://julialang.org/downloads/plotting.html? Are there objections to listing all of the packages mentioned in my original comment above? Maybe we can define criteria for inclusion there (e.g. registered packages only)?
2. We need an overview document in a canonical, easily findable/guessable location, explaining the different approaches each package took, with brief (illustrated!) examples on how to use each one. There are already some attempts to provide this overview, but they mostly live in mailing list threads, slideshows in third-party sites, or blog posts. The only one that I know of that attempts to build a community resource is the Plotting chapter of the "Introducing Julia" wikibook, so far written by @cormullion. I'd be in favor of converging our documentation/overview efforts there, which would have the advantage of a quick and painless edit process, as opposed to submitting PRs to edit http://julialang.org/downloads/plotting.html or another such vetted process.

Thoughts?

[waldyrious]

Thanks @cormullion for attempting to move this forward in JuliaLang/www_old.julialang.org#258. From the responses there by @ihnorton and @ViralBShah, it seems clear that the proposal for point 1 in my previous comment (using http://julialang.org/downloads/plotting.html as the central listing of plotting packages) is not gonna fly. Maybe the wikibooks page could be used instead?

[cormullion]

@waldyrious yes it was closed without discussion. Still, text is easily recycled. :)

[wildart]

We could start by designing an organization frontend similar to http://juliastats.github.io/. I also would like to see a example portfolio of useful cases implemented in different plotting libraries (similar to a matplotlib website). Such portfolio would definitely give a boost to an opinion exchange, and help users master visualizations in Julia.
As for organization, let it be. If developer would not have enough time to maintain package code, there always will be organization with a community potentially available to maintain code.

[lobingera]

I initially disagreed on having a plotting organization (this manpower thing), but it really looks like from the documentation perspective it makes more and more sense.

This morning i read the comments to the PR for the download/plotting webpage and somehow agreed, somehow disagreed - especially on this 'best options' as both Gadfly and PyPlot are in development, also.
However, i tried to start at the landing page julialang.org and followed to collect information about plotting (like a new user would do).
It takes a while before plotting is mentioned (gadfly in ijulia), i'm missing an early comment that plotting isn't part of the language, but a package you load upon request. And some reasoning, why there isn't a single, standard plotting package available...
My next try was the docu link which brings me to the documentation which also suprisingly doesn't mention plotting.
Community - as we know has no link to plotting.

[wildart]

I advocate for starting an organization at least as a discussion place (BTW, we are discussing general plotting issues in Julia in a Gadfly issue) and for a documentation storage. There is no harm in it.

[Sisyphuss]

I have an idea. I don't know whether it is appropriate.
As "JuliaStats" unites all related packages in the Github, we can do the same thing: unite all plot-related packages in the Github. For discussion, it can thus be centralized. For new adopters, they can find the information more easily.

ADD: It seems that it has been mentioned by @wildart 18 hours ago.

[tbreloff]

I took the plunge and created an org. I also created one repo which could hold this theoretical documentation that we've been talking about, and an issue where we could continue this conversation: https://github.com/JuliaPlot/juliaplot_docs/issues/1.

I don't think anyone should move ownership of existing plotting packages here (yet)... right now lets just consolidate wikis/tutorials, and maybe we can link to it from the julia documentation.

[Sisyphuss]

Great @tbreloff ! Now at least, we have a place to discuss and experiment with different ideas/interfaces.

[waldyrious]

Thanks @tbreloff. Let's continue this discussion over there, as it's a more appropriate venue. Everyone please add your comments in existing or new issues at https://github.com/JuliaPlot/juliaplot_docs/issues.