Allow working with Renderers interactively outside the notebook

[philippjfr]

Adds show methods for the bokeh and matplotlib renderers and an interactive option for the matplotlib renderer. Addresses #1126. @thoth291 would you mind testing if this works for you?

Here's a small example:

import holoviews as hv
import holoviews.plotting.mpl

r = hv.Store.renderers['matplotlib'].instance(interactive=True)

curve = hv.Curve(range(10)))
r.show(curve)

[jbednar]

Looks good to me. I guess you've tested that there aren't problems with getting the wrong backend in the notebook after this?

[philippjfr]

Need to remove this.

[jbednar]

In the bokeh backend??

[philippjfr]

Right, hence why I need to remove it.

[thoth291]

@philippjfr , I'm willing to test - but how should I get it installed using conda?
I've tried ioam/label/dev as I normally do - but that seems to be the wrong branch

[philippjfr]

Don't necessarily recommend this but you could:

conda uninstall holoviews
pip install https://github.com/ioam/holoviews/archive/mpl_interactive.zip

Then later you can switch back to a conda based dev release with:

pip uninstall holoviews
conda install -c ioam/label/dev holoviews

[jbednar]

But note that using the git version of holoviews isn't particularly difficult; just conda install as normal then clone this repository, do git checkout mpl_interactive, and do python setup.py develop in it. After that, do import holoviews ; print(holoviews.__file__) ; print(holoviews.__version__) to make sure you're using the git version, and you should be good to go.

[jlstevens]

If a user wants to go back to the conda versions (dev or otherwise) after running python setup.py develop in a git repo, what should they do? Simply delete the git repo? Or does a new conda install override any previous run of python setup.py develop?

I'm sure I have dealt with this myself before but either I'm having a mental blank or the answer isn't totally obvious. :-)

[jbednar]

Deleting the git repo is sufficient. develop just adds a path file, and Python moves on to the next item in the path (presumably the conda envt) when the git repo isn't found.

[thoth291]

thanks. I did this

import holoviews as hv
import holoviews.plotting.mpl

r = hv.Store.renderers['matplotlib'].instance(interactive=True)

curve = hv.Curve(range(10))
r.show(curve)

from numpy import arange
image = hv.Image(arange(1200).reshape([30,40]))
r.show(image)

And got these figures with wrong aspect for the figure:

And also these warnings (which I assume have little to do with you?):

/anaconda2/lib/python2.7/site-packages/matplotlib/axes/_base.py:1215: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if aspect == 'normal':
/anaconda2/lib/python2.7/site-packages/matplotlib/axes/_base.py:1220: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aspect in ('equal', 'auto'):

P.S. Is there any hold feature to use multiple figure windows at once?

[philippjfr]

I usually end up deleting files in the envs site-packages manually, because I don't trust the interaction between pip/conda/setuptools.

[philippjfr]

And got these figures with wrong aspect for the figure:

What's wrong about the aspect? In both examples the axis ranges are of equal range so even if you enabled aspect='equal' (which is the default for Image) the aspect is expected to be square.

And also these warnings (which I assume have little to do with you?):

Right, those are annoying me quite a bit now, they've been fixed in matplotlib master a while back but for some reason those fixes didn't make it into 2.0. I'll have another look at avoiding them.

[philippjfr]

P.S. Is there any hold feature to use multiple figure windows at once?

Does running a show always close the previous figure right now?

[thoth291]

Does running a show always close the previous figure right now?

No it doesn't close it - once you close the first one - second appears.

What's wrong about the aspect?

Wrong aspect for the figure window - not for the plot in it.
It's weird cause if you do this:

from matplotlib import pyplot
pyplot.plot(range(10))
pyplot.show()

You get this:

I guess it's the way how matplotlib handles the figure size and plot aspect in that figure...

[thoth291]

Essentially figure should inherit the plot's aspect ratio.
And currently I don't see it does...

[philippjfr]

Essentially figure should inherit the plot's aspect ratio.

Thanks for clarifying I'll have to look into that.

[philippjfr]

P.S. Is there any hold feature to use multiple figure windows at once?

I'll look into that as well. I'm probably revealing my unfamiliarity with interactive use of matplotlib here, but is there any reason why it shouldn't just always "hold" figures letting you display multiple at once?

[thoth291]

I think it's due to the defaults from Matlab...
Some Matlab (:alien:) users tend to use plots as a breakpoints for QC the output...

[philippjfr]

In interactive mode you don't actually need to call show, so I've changed things slightly. Therefore when in interactive mode MPLRenderer.show is now just an alias for MPLRenderer.get_plot. This also means that in interactive mode you can update the plot by calling update on it:

import numpy as np
import holoviews as hv
import holoviews.plotting.mpl

r = hv.Store.renderers['matplotlib'].instance(interactive=True)

curve = hv.Curve(range(10))
r.show(curve)

image = hv.HoloMap({i: hv.Image(np.random.rand(10,10)) for i in range(10)})
plot = r.show(image)
plot.update((2,))

What I have noticed is that you can't paste the script into ipython and execute it in one go. Each call to show must be executed separately for it to keep both plots open. Ignore that, fixed.

[philippjfr]

Here's what it looks and works like for me now, no aspect issue but I'm using the Qt5 matplotlib backend. Could you check which GUI backend you're running?

[thoth291]

I tested it.
It works for me with qt4 only - it's might be something in my setup...
And it stopped working when I call it outside of ipython - let's say using python test.py

What I did:
put a copy of your script in test.py.
run it

python test.py

Got only aspect warnings.

Then started ipython and run it like this:

%run test.py

Again, got only warnings and no figures.

Then did this in the new ipython kernel:

%matplotlib qt4
%run test.py

And it worked (except for aspect issue).

So I would say that the version before this one worked better for me (cause it worked outside of ipython).

@philippjfr , may I add one more thought into consideration?
The idea of holoviews to support interactive mode is that one can write code once and enjoy it regardless whether he/she runs it in the notebook or in as standalone app.
Recall matplotlib - which code works in both Jupyter and standalone mode - without a single change in the code.
I would say that hv.notebook_extension() should set the defaults somehow and understand whether it runs from Jupyter or as a standalone so that all the rest calls would be rendered accordingly....

[philippjfr]

It works for me with qt4 only - it's might be something in my setup...
And it stopped working when I call it outside of ipython - let's say using python test.py

Sorry we've been misunderstanding each other, interactive mode is meant for working interactively from an IPython console. I've just pushed another change that should let you work from scripts. Here's a small example, you simply call MPLRenderer.show with a list of objects you want to display:

import numpy as np
import holoviews as hv
import holoviews.plotting.mpl

r = hv.Store.renderers['matplotlib']

curve = hv.Curve(range(10))
img = hv.Image(np.random.rand(10,10))
r.show([curve, img, curve+img])

The idea of holoviews to support interactive mode is that one can write code once and enjoy it regardless whether he/she runs it in the notebook or in as standalone app.

The main difference is that holoviews objects themselves are not plots they are simply your data with some metadata. The notebook provides hooks to automatically transform the objects containing your data into a visual representation. Outside the notebook context you unfortunately need a bit of boilerplate to turn the objects into plots. In matplotlib you also have to initialize your backend (e.g. %matplotlib inline in the notebook), and when working from a script afaik I know you have to explicitly call plt.show().

Let me know what you think of the new approach above, I do want to make the transition between notebook and scripts as smooth as possible, without adding unnecessary boilerplate.

[philippjfr]

Note I haven't figured out the aspect issues yet, so you saw those in Qt4?

[philippjfr]

I would say that hv.notebook_extension() should set the defaults somehow and understand whether it runs from Jupyter or as a standalone so that all the rest calls would be rendered accordingly....

I did consider this when originally implementing the notebook extension and perhaps we should consider finding a better solution here, specifically around importing the backend. At minimum we might be able to let hv.notebook_extension work outside the notebook and do nothing except loading the plotting backends. Then you don't have to run import holoviews.plotting.mpl explicitly. You would still have to get a handle on the matplotlib renderer and call show though.

[jbednar]

You would still have to get a handle on the matplotlib renderer and call show though.

Unless notebook_extension returns the default renderer, in which case someone could save that whether or not they are in the notebook...

[philippjfr]

Unless notebook_extension returns the default renderer, in which case someone could save that whether or not they are in the notebook...

Yes, that makes sense, I thought it might return all initialized renderers, which would be awkward but returning the active renderer makes sense. It does clutter all cells that end in hv.notebook_extension(), which is probably the case in most notebook, with even more output though. Wondering whether making returning the renderer conditional on whether you are in the notebook. @jlstevens did we ever establish a clean way to determine whether you're in the notebook?

[thoth291]

Why don't you let people to set some env. variables to control default bahaviour. That would include back-end, Jupyter v.s. standalone, output format and etc. Then notebook_extension would set what was explicitly given. I would then suggest default to be for Jupyter and suggest users how set everything for standalone usage.

[jbednar]

I'd rather not use envt variables if at all possible; that's a clunky way to do it.

Good point about the values that would get printed in the notebook; output_notebook does indeed often end a cell at present, mainly because of the logo/message it prints (which I've voted to get rid of anyway :-). In that case, you're right; probably better not to return anything by default. I suppose one could have an argument that would enable such returns, but that too is a bit clunky.

[jlstevens]

did we ever establish a clean way to determine whether you're in the notebook?

A clean way? I don't consider checking the type of sys.stdout particularly clean.

I don't think using notebook_extension outside of notebooks makes much sense as it would be misnamed, if nothing else. Auto-detecting whether you are in the notebook or not is something the Jupyter devs fight against, so I wouldn't recommend that.

What I would be open to is an equivalent for pure Python users, e.g hv.load_backends() that does what we need here. Maybe it could be written such that notebook_extension uses the same machinery/is an extension of it. In addition, if it is idempotent then there would be no harm including in in either context. In other words maybe, we can make load_backends redundant in a notebook context (does nothing) and maybe we can find a way to make notebook_extension do nothing in a non notebook-context (i.e as long as IPython is available but notebooks aren't being used).

[thoth291]

I just thought a little bit more about it and actually would say that adding a global feature like hv.rc.standalone=False which (would be default) customize the behavior similar to %matplotlib inline will work just fine for many people.
For example I could write modules which are producing holoviews objects and then use them like that:

##%%writefile test.py
import holoviews as hv
from my_hv_objects import myDataCurve, myDataImage
if __name__=="__main__":
    hv.rc.standalone=True
    myDataCurve()
    myDataImage()

And for interactive/html

##Jupyter
from test import myDataCurve, myDataImage
myDataCurve()
myDataImage()

But for standalone simple

python test.py

or in ipython

%run test.py

would do the trick

[thoth291]

Another idea is to create nbconvert plugin which would add all necessary hooks to the script prior to saving it as python so that generated script would work standalone.

[philippjfr]

Another idea is to create nbconvert plugin which would add all necessary hooks to the script prior to saving it as python so that generated script would work standalone.

If you're willing to go down that route it seems to me like you would actually be better served by generating HTML reports from your notebooks directly using nbconvert. How does that sound?

[thoth291]

@philippjfr I can generate HTML reports right now - but that's not sufficient. What I'm trying to help my users with is this:

1. User does it's work in Notebook
2. Once the script is established the user creates from it a normal standalone script
3. Then deploys it to the other user and let them generate all the plots and etc.

The thing is to minimize manual work need to be done from step 1 to step 2.

Currently users use matplotlib most of the time and then just copy-paste all the stuff to the standalone script. If they have any bokeh or plotly - they keep it for further R&D in the notebook - but drop them from converting to standalone script or reimplement some pieces of it using matplotlib.

I can instantly see how useful holoviews can be for this workflow.
Do I see it wrong?

[jbednar]

Interesting. That's not a workflow that I would use personally, because to me a notebook is much more of a shareable thing than a script is, because it can include interleaved Python, descriptions, and output, where it's possible to tell a story that other people can read and understand. And here HoloViews helps greatly in making a notebook that can tell a story, as opposed to a notebook crammed with unreadable plotting code -- it should all be readable!

But maybe I'm missing the point of the standalone script, when used with HoloViews.

[thoth291]

I agree with you on notebooks - but that's not all what people do.
Telling a story is only 10% of the work. Develop a solution - is 90% of it. You don't need to tell a story when being asked for the answer.
You can't really deploy a notebook - as it can't do many things which one would find useful in production world. I understand holoviews better now - but I don't see it as a replacement for my ugly matplotlib code in the notebook. What we are doing right now - is essentially old trick - wrap all your plotting into convinience function plotMe and use it everywhere in the notebook. Works better than you might think - up to the point when someone else have to deal with that notebook - then even author can't remember why he/she implemented the plotting this way...

Matplotlib code can be reused in standalone world - holoviews in it existing form can't.
And coming back to my workflow - I wouldn't throw away notebook - I would keep it for further advances in R&D but would use my standalone script for routine work.

Essentially we are using Jupyter not for final presentation or report only. We are using it in order to develop ideas - but we are not deploying through it - it's inefficient at least in the sense that other users would need to know Jupyter, start the server with all dependencies and be able to run it with their data. Instead we build a standalone script which looks and feels like any other C/C++/Fortran program and users don't even notice that they run python....

Regardless of all what I have said - holoviews is very promising product - but limiting it to Jupyter only or complicate transition to standalone is waste of potential.

[thoth291]

I've been dreaming about holoviews all these years - just never realized that.
So far I spent 10 times less time to build 85% of the plot using holoviews comparing to other tools.
Moreover, most of the time the quality of the final widget is way better what I could do with my matplotlib+ipywidgets or bokeh.
But the last 15% collapsing the advantage of holoviews into somewhat random flags and properties in the %%opts magic. Checkout all the issues I reported here.

I'll share with you one quite standard notebook soon which is generated based on one of my user's Matlab code - and I honestly don't even think that it's possible to be done in holoviews....
But I figured it out using matplotlib (the bokeh implementation seems to be more of a dream).

I wonder how difficult it would be to build a holoviews code generator based on existing matplotlib figure?

[philippjfr]

With the release of 1.7 coming up I'm going to suggest we merge this as is and can consider how we can make it easier yet. For now this makes it possible to use the matplotlib backend from scripts and in interactive mode in an IPython terminal, which wasn't allowed before. Since bokeh ships facilities to open a bokeh plot in a browser I'm not going to duplicate this in a BokehRenderer.show here and have removed that part again.

I'll keep the corresponding issue open to keep the discussion open and at minimum remind us to add documentation.

[jlstevens]

I agree with Philipp.

We may still find a better approach but this PR does go a long way towards the requested functionality. I do think it will really help with any HoloViews usage outside the notebook.

Merging.

[thoth291]

Thanks!
I tested it and can confirm that the code

import numpy as np
import holoviews as hv
import holoviews.plotting.mpl
print hv.__version__

r = hv.Store.renderers['matplotlib']

curve = hv.Curve(range(10))
img = hv.Image(np.random.rand(10,10))
_=r.show([curve, img, curve+img])

Generates almost the same plots with no change to the code!
Well done!

Two comments:

1. Why layout is shifted in the notebook (notice the shift to the right from one row to another)?
   

2. Is this warning too important to be ignored and should I expect it to disappear?:

/anaconda/lib/python2.7/site-packages/matplotlib/axes/_base.py:1292: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if aspect == 'normal':
/anaconda/lib/python2.7/site-packages/matplotlib/axes/_base.py:1297: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aspect in ('equal', 'auto'):

P.S. I'm using matplotlib=2.0.0

Thanks!