Transition to Plots.jl

[SimonDanisch]

Considering that MakiE is designed quite differently from Plots.jl, any transition to Plots.jl will
basically mean replacing Plots.jl. I don't see how to slowly incorporate ideas from MakiE into Plots.jl without a huge amount of work. I think we're at a good time to still make drastic much needed changes, so we shouldn't try to be conservative here.

So I propose to let MakiE live as it's own package for a while until we can ensure, that all features from Plots.jl are well covered.
Then we can think about renaming MakiE, make a PR to Plots.jl to replace the internals with MakiE or simply deprecate Plots.jl and suggest to move to MakiE.

This would look something like this:

1. Pre release MakiE (should happen today)
2. Implement some more high level recipes from Plots.jl
3. Introduce a compatibility layer for PlotsRecipeBase to see if MakiE support the whole range of features used in packages
4. factor out MakiE Plotsbase into an abstract plotting package with no dependencies
5. Create backend packages
6. Refactor into packages - replace/rename Plots.jl with new internals

Other backends

I'm inclined to implement a reference backend different from GLVisualize with Cairo. I already have some code for this from my experiments in Visualize.jl.
Cairo is completely orthogonal to GLVisualize and offers missing bits and pieces like PDF/SVG export.
The orthogonality seems to be an important quality for different backends to make the choice of backends clearer to the user.

Since GR is the favorite Plots.jl backend, it would also make sense to start with GR as a reference implementation.
It's less orthogonal and I'm not as familiar with the API, though.
But if @jheinen is on board and willing to help me out, this could also be a great start!

cc: @daschw @mkborregaard @piever @ChrisRackauckas @Evizero @ViralBShah

[ChrisRackauckas]

Then we can think about renaming MakiE, make a PR to Plots.jl to replace the internals with MakiE or simply deprecate Plots.jl and suggest to move to MakiE.

I'm all for this. If it fixes the first-time-to-plot issues and formalizes the backend handling to get rid of the conditional dependency issues, that's worth a fresh start IMO

To me that roadmap makes sense.

[piever]

The plan makes sense until point 5, but I think before deprecating Plots.jl there are several things that would need to happen and may not be trivial to get:

• How fast will the time to first plot be with MakiE? As of now it's not super fast either (at least on my machine) so I'm not sure that's a selling point, but maybe it can be improved (I guess it depends on the time to "warm up" GLVisualize, so I suppose it'd be very fast with say GR given that there you download the binary)

• We still need to make sure that the compatibility layer can exist between old recipes and MakiE (and see how would be the transition exactly)

• Exporting: Plots exporting is actually very rich (pdf, svg, eps, png but also tex or even html that could be embedded in a website, with plotly). Of course a Cairo or GR backend could give the
  nice vectorial format, but html and tex would no longer be an option. We'd need to make sure that this new "low-level" interface with the backend can be extended to Plotly and PGFPlots.

[SimonDanisch]

How fast will the time to first plot be with MakiE?

Precompilation should work better, so indeed with GR it should be much faster. I'm also experimenting with julia-static, but haven't gotten any answers from e.g. @vtjnash about some of the problems I run into.

We still need to make sure that the compatibility layer can exist between old recipes and MakiE

that's point 3, isn't it?

interface with the backend can be extended to Plotly and PGFPlots.

I'd love to just have our own webgl backend ;) If you don't use webgl, you might as well just embed an svg in html! But Plotly and PGFPlots would be great! I guess anyone can help with those once we have a reference backend!

[mkborregaard]

Hi Simon,
I'm positive about this project. I think there's no doubt that there are issues with Plots that should be adressed with a refactoring of the code, and that you've identified these correctly: 1) issues with world age and precompilation due to eval-ing in conditional dependencies; 2) limited support for interactivity in that existing possibilities (in the backends) of updating plots, rather than redrawing the plot en bloc, aren't leveraged; 3) limited support for the GPU, and possibly 4) some loss of fine-grained plot control in the deliberately simple interface.

The reason I'm invested in Plots is that it has 1) the recipes, allowing all plotting to be based on methods for custom types by essentially removing the need for packages to depend on a plotting library; 2) the terse, smart syntax, and 3) the ability to split backend from interface/syntax.

It looks like a reimplementation of Plots based on MakiE could resolve the problems while keeping the advantages. MakiE appears modern, and I agree that now, pre-julia-1.0, is a great time to allow some drastic changes to Plots internals.

With regard to your roadmap, I agree completely that it would make sense to have two development paths (Plots.jl and MakiE.jl) until we have shown that MakiE can attend feature parity with current Plots. There's been a lot of development on Plots (there's close to a new PR merged every day and have been for a long time, though many PRs are small), and many of the fixes and improvements are fairly subtle and based on users making literally hundreds of thousands of real-world plots with the package. So the question is, how do we ensure that all of that experience isn't lost?

I think one good approach could be to go through all the PRs from the last year and ensure that the functionality and corrections they've added is reflected in the current examples (doubling as tests). We've become better at doing that, but there was a long period where that wasn't in place. We'll add some new examples to make sure this is so, and those examples could then serve as a development target for MakiE - essentially, when they pass we'll know we're at feature parity.

With regards to deprecating Plots vs renaming the new MakiE to Plots and replacing the library when it's ready I don't have any strong feelings about this, although Plots is the better name IMHO. If MakiE has feature parity and can be used with the same syntax and backends, while resolving the issues, I see no reason it shouldn't replace the current Plots library.

One thing you might consider now is whether to put the library in the JuliaPlots org? I think we could still set it up so only you have owner and push rights to the Repo. I think that would help to put it on the radar for the other members of the org if you'll like us to engage with the project, and could also make it easier to coordinate things.

Instead of moving things over from Plots we could consider parceling out some of the functionality that could be used unchanged in MakiE in a separate package, which Plots and MakiE would then both depend on. I don't have a good enough overview of MakiE to identify exactly what those things are, but could be e.g. the recipes and arg files, much of the backends etc.

[tknopp]

I am looking forward to the Cairo backend. This will smoothly integrate with Gtk.jl.

[daschw]

👍 to everything @mkborregaard wrote.
I'm also positive about this project, but I am not sure if I understand the relations between Plots, PlotsBase, MakiE, backends, ... correctly. Here are some of my assumptions that I am not completely sure of. Could you please verify them or correct me if I misread this:

• MakiE will be split up into PlotsBase and MakiE (which will then be something like a PlotsBase backend?)
• The aim is to preserve current Plots syntax. So Plots.jl will be rewritten to something that translates the current Plots syntax to the PlotsBase (currently MakiE/plotsbase) internals. This could also coexist with something like GGPlots.jl
• Other backends (like GR) will be implemented for PlotsBase

If I understood this correctly this sounds a lot like @mkborregaard's suggestion here:

ChosenInterfacePackage-> BasePlotSpecification->ChosenOutputPackage

and I do like that a lot :-)

[juliohm]

The reason I'm invested in Plots is that it has 1) the recipes, allowing all plotting to be based on methods for custom types by essentially removing the need for packages to depend on a plotting library; 2) the terse, smart syntax, and 3) the ability to split backend from interface/syntax.

I want to stress @mkborregaard words here. These Plots.jl features are quite important. No one wants to go back to ugly syntax or to a scenario in which recipes don't work well.

[SimonDanisch]

I want to stress @mkborregaard words here.

Yes, that's the plan! Recipes will actually be much more first class and probably more flexible.

@daschw this is pretty much the plan :)

ChosenInterfacePackage-> BasePlotSpecification->ChosenOutputPackage

Yes, I like this too! Since the higher level syntax itself will basically just be a bunch of recipes, switching out the interface should become more natural!

I think one good approach could be to go through all the PRs from the last year and ensure that the functionality and corrections they've added is reflected in the current examples (doubling as tests).

I was sure that I already answered this. It would indeed be lovely, if we can grow the plots.jl testing suite before making a switch! Maybe that's something the community can do during the next month! We should announce this somewhere, that everyone who had a problem with Plots.jl should open a pr with a small code snippet reproducing that problem!

[mkborregaard]

We should announce this somewhere, that everyone who had a problem with Plots.jl should open a pr with a small code snippet reproducing that problem!

No need for that, IMHO. Everybody who approaches us with a real issue are suggested to open an issue with an MWE. We currently have 219 open issues, and the list is quite up-to-date – reading through those should give a very good impression of the shortcomings that would be relevant to address.

What I meant was to ensure all recent fixes and enhancements were covered by tests. I'll open an issue to track this.

[SimonDanisch]

well, making a testsuite out of it ( prs, issues, anything) and not having a bunch of issues is exactly what i meant to call for ;)

[mkborregaard]

Do you mean making a testsuite out of things that aren't fixed yet? Nice idea, but tests would have to work different than they currently do in Plots (they compare plots to a reference image).

[SimonDanisch]

out of things that aren't fixed yet

I don't really understand what you mean...
It would just be nice, if Plots has something through which I can run quickly to verify feature completeness and that there are no bugs!

I don't really care if the source of those tests are issues, prs or whatever ;) It also shouldn't matter much, whether they're against a reference image, or if I need to run them in Plots.jl and MakiE and compare them myself. It's just important for me, that these are at a single place and quickly to execute, because it will be a painful process even under perfect circumstances ;)

[mkborregaard]

This: https://github.com/JuliaPlots/Plots.jl/blob/master/test/runtests.jl#L3-L16

It runs all the examples (https://github.com/JuliaPlots/Plots.jl/blob/master/src/examples.jl) and compares the output to reference images stored here: https://github.com/JuliaPlots/PlotReferenceImages.jl/tree/master/Plots

When those tests pass, there should be feature completeness (well, StatPlots, PlotRecipes and all package recipes should work as well for feature completeness, we'll work on building a similar system for them).

(I first thought you were talking about tests-first programming.)

[oschulz]

It would be nice if, like in Plots, there would be a lightweight package like RecipesBase - so that packages don't have to require MakiE itself to define recipes.

[SimonDanisch]

Whenever there will be a nice way to define recipes with Makie, it will be as lightweight and future rich as in RecipesBase :)