Add GR to REQUIRE

[davidanthoff]

With this, the out-of-the-box experience for Plots.jl should always be GR.

There are lots of things that could be improved, like: a) add a min bound on GR (@jheinen do you have a suggestion what I should put in here as the min GR version) and b) get rid of the conditional load code for the GR backend.

If @jheinen helps me with a) I will update this PR before we merge it. I'll leave b) to someone else.

[mkborregaard]

I realize this is a major breach of what has been Plots philosophy so far – people prefer different backends, and there's no a priori reason Plots should special-case anyone of them. @ChrisRackauckas has argued in the past for always having Plotly be the first choice (because of GR terminal's behaviour on Windows: it cannot resize manually, and the window doesn't reopen if you close it; this should hopefully improve "soon").
There's also the issue that GR has a binary dependency.

Still, I support it. A lot of people are requesting a more seamless experience for beginners, associated with more people teaching courses in Julia. And in my experience GR is the smoothest-working backend.

This should be discussed broadly. @jheinen @SimonDanisch @daschw @apalugniok @piever @wkearn ?

[ChrisRackauckas]

These days, except for GR on Windows in the REPL, GR is a great go-to. Even on Windows in Jupyter and Juno (I'd assume VSCode too) it's great. I think privileging it to give an easier beginning experience is a good idea.

[piever]

I'm also in favor (and can confirm that GR works smoothly on VSCode too). I'm not sure whether one gets much of a speed up (see this coment), but we could consider adding a import GR at the top level and removing the lazy loading.

[pkofod]

I'm in favor as well - on the condition tha jheinen is in favor. By that I of course mean that it is in his plans to continue to help plots work well with gr in the future .

[jheinen]

• We will continue to support the GR Plots backend
• The next GR release will contain a GKS QtTerm, which will improve the terminal behaviour on Windows, too. Right now, it's "only" a deployment problem.

@mkborregaard : What exactly do you mean with "There's also the issue that GR has a binary dependency"? I think we can't change the current delivery process - there are so many distribution options, e.g., on macOS, we have MacPorts, Homebrew, Conda-Forge and others ...

[ChrisRackauckas]

GR's binary does seem to be the safest though.

[jheinen]

The GR run-time for Windows is build with MinGW-w64. If we "find" a suitable Qt5 package, we could provide a better terminal for Windows soon. sciapp/gr@4edb0f0 works already fine on macOS and Linux (using Qt 5.9).

[apalugniok]

I'm also in favor, GR has been working well for me for quite a while.

[mkborregaard]

Just that binary dependencies can be more tricky to install. I've never had issues installing GR but in the past there were sometimes people who had trouble.

[ChrisRackauckas]

I agree with being weary of binaries, but at this point GR is the only binary I know of that will install on crazy clusters with old GCC versions, and newest versions of Linux, and Windows, etc. Here's one issue which GR could solve that others couldn't:

#519

@jheinen has done a great job with it and I think that, given we've had a few years to mess with it, we should evaluate its robustness on its own. For example, when was the last installation issue filed to Plots about GR? There have been tons for PyPlot (because of its >v2.3.1 madness), but I don't recall one on GR lately.

[mkborregaard]

No, I agree. As I said I am in favour. I'm looking at the optional loading bit now.

[mkborregaard]

I think the Travis failure is because these two lines should be commented out - would you try that @davidanthoff ?
https://github.com/JuliaPlots/Plots.jl/blob/master/test/travis_commands.jl#L4-L5

[davidanthoff]

@jheinen Any recommendation on the lower bound for GR?

[daschw]

Don't get me wrong, GR has been my favourite backend for a long time now and I am 100% in favour of GR staying the default backend. However, I am a little sceptical about adding this dependency. @ChrisRackauckas and @piever have already reported that getting rid of the conditional load won't improve performance. It even seems that it would increase the time for using Plots.

So what do we gain from adding this dependency? Users, who want to have GR as the default backend no longer have to do Pkg.add("GR"). Is this really such a big improvement to an out-of-the-box working plotting environment or am I missing something else? GR is even pre-installed on juliabox so it is the default there.

On the other hand, adding the dependency would force users who just want to use a different backend to install GR. I'm not really sure, if that could be such a big porblem to anyone, though.

That said, I'm not really against this, I just wanted to share my thoughts.

[mkborregaard]

I think those are all really good points. Just to answer the question of "why?": It's for those users who don't know to type Pkg.add("GR") after installing Plots. The idea is that students and other naive users will get the best possible experience by default.

[jtravs]

As an outsider (but one who uses Julia to teach in a physics classroom), here I only only comment that GR is currently completely unusable on windows (as mentioned above). Surely this should wait until the promised fixes to the windows terminal are actually a reality? It seems to me that until then other back ends would be more reliable?

[mkborregaard]

"completely unusable" refers to the fact that the windows doesn't resize and shouldn't be closed manually?

[jtravs]

Yes, in almost any interactive plotting one usually wants to close plots and open others when you explore data. The only solution with current GR (AFAIK) is to restart Julia. Resizing is also very frustrating, as is the general lack of interactivity. I don't have any "skin" in this game, as I will continue using what works for me, but I did want to raise a counterpoint here to the general enthusiasm as maybe it is not based on classroom experience (on windows) but on developer friendly machines.

[ChrisRackauckas]

FWIW it does work fine on Windows in Juno and Jupyter though, so it's just the GUI window.

[jheinen]

@davidanthoff : v0.25.0

v0.26.0 coming mid Feb

[mkborregaard]

I think the origin of this was @davidanthoff 's experience that the current no-backends-installed default, plotly, has lots of issues in Jupyter notebooks (which seems to stem from the backend, not from Plots). An alternative we discussed was to have fmt = :png the default in notebooks, which would loose the interactivity (unless you turned it on) but would improve the beginner's experience with Plots+PyPlot+IJulia. @davidanthoff do you see that as an alternative to this?

[mkborregaard]

Is v0.26.0 be the one that brings the terminal improvements?

[jheinen]

Yes - that's the plan :-)

[davidanthoff]

@daschw The impetus for this PR is the following scenario:

User installs Plots. Opens an IJulia noteook, plots something (it will use the plotly backend), saves the notebook, opens it again and sees lots of javascript errors. That is in my mind like one of the top 2-3 scenarios that new users will do and I feel strongly that the "default" plotting story for julia just has to get that right, i.e. bug free. I think even starting to talk about "oh, you just need a different backend that you first need to install" is a no-starter for novice users: they don't have to do that on something like R or Python, and will just turn around and say julia is not ready for them.

Things also don't work with jupyterlab with the plotly default backend because it disables javascript in emitted HTML, which the plotly backend relies on.

This PR seemed the easiest way to achieve that, but of course it would be great if there were some other ways to achieve this.

I was not aware of the issues with GR on the windows REPL. I just tried it, and I think my vote is that these issues are less bad than the default IJulia experience right now, so I'd be in favor of living with that issue until GR v0.26.0 is out.

@mkborregaard I don't understand how fmt = :png would work with the default plotly backend in IJulia? To generate a png you need to run the plotly javascript code, and that seems to be exactly the problem?

[daschw]

Good points @davidanthoff 👍 from me

[mkborregaard]

It would run the javascript code, but not emit it as output to the cell. Instead it would emit a png image which would be embedded in the notebook, thus persistent (but non-interactive)

[mkborregaard]

I think if v0.26.0 comes mid-Feb we could perhaps just wait for that? I've looked at some ways of bypassing the optional GR loading, but as @daschw said it doesn't create a dramatic improvement in time-to-first-plot - I will be checking that more though.

[davidanthoff]

It would run the javascript code, but not emit it as output to the cell. Instead it would emit a png image which would be embedded in the notebook, thus persistent (but non-interactive)

How would it do that? I don't think there is an Jupyter API that allows javascript code that is transmitted via the HTML MIME type to add an additional representation of the cell output as another MIME type, right? Plus, running javascript is entirely disabled in jupyterlab.

I think if v0.26.0 comes mid-Feb we could perhaps just wait for that?

I don't feel strongly about it either way, another 10 days really won't matter much, and if the REPL Windows story then also works, it would obviously be nicer.

[mkborregaard]

How would it do that?

Just realized the plotly backend (as opposed to the plotlyjs one) doesn't support fmt = :png anyway.

[jlperla]

Just please reconsider this decision in light of the evidence that PlotlyJS.jl now seems to have a solution to the dissappearing plots issue. While that was a deal killer for a plotly based default backend, it may not now. If the plotly backend can do the same tricks, then I think GR should be reconsidered.

[cstjean]

I don't have an opinion on default backends, but I'll note that on Ubuntu+IJulia+Firefox, GR has some scaling issues. Eg.

scatter(randn(10000))

makes my browser struggle, and the text boxes have input lag. Notebook size is 3 MB. Going to 100K points, I have to kill the browser (on a reasonably powerful desktop). In contrast, because it merely displays a PNG, PyPlot can handle a million points without a problem, and file size / memory footprint stay small.

[ChrisRackauckas]

GR has some scaling issues.

No, you didn't set fmt = :png

[cstjean]

TIL, thank you!

[davidanthoff]

Well, maybe PNG should be the default in that case... I really think we need to get things to a state where people don't have to know about things like fmt=:png to just get a good experience...

[jheinen]

@cstjean : If you want to plot millions of points in IJulia using in SVG, million of points have to be rendered by your browser resulting in an unresponsive behavior. In a future release, I plan to reduce the data before plotting. For now, you have to switch to a suitable output format, e.g. PNG.

[piever]

Well, maybe PNG should be the default in that case...

@davidanthoff, I've made a PR: #1394, but there are scaling issues due to GR high dpi. I think it's a high priority to fix those (they also affect how gr png display in Gtk and QML: see here). I do not have the expertise to fix those (I barely understand what they are about), but once they are fixed we can merge the PR and switch to a png based solution in the notebook.

[mkborregaard]

I'll just note that I'm against the png-based solution. There's a lot of focus on having something that works seemlessly for beginners. But that is not the only priority here. We also want a nice library, and the png format discards all interactivity, which is really nice to have in notebooks. Also, using a notebook to plot millions of points and then experiencing a slowdown is not a typical beginner's issue IMHO.
I think a short-term solution is to solve the persistence issue with plotly plots in notebooks. I think the new JSTerm should be the default for GR once that comes out.

[piever]

Agreed, we should maybe advertise OnlineStats recipes a bit more when plotting large data (the scatter plot with millions of points is rarely the most informative visualization). I've opened the png PR but I don't think it should be merged (not without unanimity at least and a fix to these dpi issues) and I agree that an interactive solution is much nicer. What is the status of JSTerm?

[mkborregaard]

Supposed to be just around the corner

[cstjean]

Also, using a notebook to plot millions of points and then experiencing a slowdown is not a typical beginner's issue.

Millions of points crash my machine. GR has 1-2 seconds of input lag (i.e. typing "hello" takes a few seconds to show up) with only ten thousand points which is, IMHO, fairly modest. PlotlyJS seems to fare a bit better, but there's still noticeable input lag.

For comparison, I've tried Bokeh in Python, and even with a million points, it causes 0 keyboard input lag. Zooming and panning is a struggle though.

from bokeh.plotting import figure, show, output_notebook
from numpy.random import randn

output_notebook()

p = figure()
p.circle(randn(1000000), randn(1000000))
show(p)

the scatter plot with millions of points is rarely the most informative visualization

With alpha=0.01, it's a useful density plot.

We also want a nice library, and the png format discards all interactivity, which is really nice to have in notebooks

+100

[piever]

the scatter plot with millions of points is rarely the most informative visualization

With alpha=0.01, it's a useful density plot.

Fair enough, but rendering a million points with transparency is even harder and I really believe the IndexedPartition plots are a better way to go about it (see in particular the Hist example). Or, even more simply, we do support 2D kernel density estimation visualizations (with contour, contourf, heatmap and surface).

If visualizing this kind of data comes at the cost of interactivity, I'd tend to agree with @mkborregaard and favor interactivity. An intermediate solution would be, if the number of points to plot is too high (say length(d[:y]) > 100_000 to warn: "Plotting more than 100_000 points may crash your browser, consider using fmt = :png" (though, once again, we'd need to fix the scaling issues with png with GR in a notebook).

[ChrisRackauckas]

Has the new GUI been released?

[jheinen]

Let's call it a pre-release. If you are on GR master, you can use the new GKS Qt terminal by setting an environment variable (GKSwstype=gksqt) before importing Plots. Requires GR run-time version 0.29.0.post13, so you probably have to download the most recent one (ENV["GRDIR"]=""; Pkg.build("GR")).

It works quite stable (and fast) and I'm going to make it the default backend (on Linux and Windows) with the next release.

[ChrisRackauckas]

GR v0.27.0 fixes the window re-opening bug on Windows and is solid there. Is this ready to be made the an installation default backend?

[ChrisRackauckas]

@jheinen is there anything keeping it from being a full release? It seems to be very solid and I help fix people's "installation issues" in chat by having them do:

ENV["GKSwstype"]="gksqt"
ENV["GRDIR"]=""; Pkg.build("GR")

IMO that makes it seem ready.

[jheinen]

My plan was to implement the multiple window feature first, but if you think, it's (already) solid enough, we can make it the default output device for Windows and Linux ...

[ChrisRackauckas]

I'll let others weigh in, but from my experience it's a big usability enhancement (it fixes the Windows closing bug!) and I can't find issues with it. I think it's ready for a release.

[daschw]

I will add the minimal GR version in a separate PR.