Colab rendering requires reloading extension in each cell

[loopinf]

I've searched the issue using holoviews in colab.

only matplotlib rendering is possible.

do you have plan for bokeh rendering?

[jbednar]

We'd be happy for Bokeh and HoloViews to be available in Google Colaboratory, but I'm not sure there is anything we can do on our side; I was under the impression that Google would need to be the ones to set that up. But if there is anything we can do, please let us know!

[jlstevens]

I also believe that google runs a non-standard version of notebook which they've customized in a way that makes things difficult.

[philippjfr]

We can look into it a bit, it was possible to get things to work by loading the extension in the same cell as the displayed plot when I last looked at this.

That said, I believe what Google is doing with Colab is exceptionally bad for the Jupyter ecosystem. They are already producing notebook files that are non-compliant with the notebook spec, they are not following various other standards for rendering of output, and ultimately they should be contributing their improvements back to Jupyter itself rather than maintaining their own constantly diverging fork. Projects that are supported by Google directly will continue to receive attention from their devs while everyone else is excluded by the fact that their fork is not open source and cannot easily be extended.

[philippjfr]

Currently this is the only way to render plots in colab: First set this env var:

import os
import holoviews as hv

os.environ['HV_DOC_HTML'] = 'true'

Then to render a plot you have to make sure that the extension is (re)loaded in the same cell, every time you want to see a plot in that notebook:

hv.extension('bokeh')

hv.Curve([1, 2, 3])

[loopinf]

Thanks for the answer!
I'll try this.

[josephsmann]

FYI

import os
import holoviews as hv

os.environ['HV_DOC_HTML'] = 'true'
hv.extension('bokeh') 

is throwing an unsupported backend error on hv.extension('bokeh')
hv.version --> '1.12.3'
bokeh.version --> '1.0.4'

[awav]

@josephsmann , you need to update bokeh version !pip install -U bokeh. My problem is that DynamicMap doesn't appear in the output of the cell in colab. Does anybody know - is that fixable?

[blois]

Hi- Colab developer here, hoping I can lend a hand if possible.

I know Colab's output rendering can be a pain sometimes, we do try to make it seamless and honestly do want to have as few divergences as possible. By far, the largest issue is going to be the iframing of outputs. These have been mandated by our security team since the inception of the Colab project for a number of reasons, there's almost no chance of Colab moving away from them. We firmly believe that the security layer they introduce is necessary.

A bit of background, mostly before my time with the project:
Initial development of the frontend with the iframed outputs was fully intended to be upstreamed and that work can still be found at https://github.com/jupyter/colaboratory. My understanding is that the iframing of outputs was hard-rejected for compatibility reasons, but Google's security team mandated this to allow Jupyter Notebooks to be used internally. So in ~2014 efforts to upstream iframed outputs ceased and Colab became an internal Google tool.

More recently the nteract project has expressed some interest in the iframed outputs model and we're happy to assist, but my understanding is that there's pretty much no chance that JupyterLab would adopt this model because of compatibility reasons.

This was all a long way of saying that I'm sorry for the differences. My goal has been to work within our security requirements and try to make things 'just work', this is a case where things aren't just working and I'd like to see if there's something Colab can do.

Any compliance issues with the notebook files should be filed at https://github.com/googlecolab/colabtools/issues/new, we try to address these quickly (there are no outstanding issues here that I'm aware of).

[jbednar]

Thanks for that background! We'll file a colab issue with the details.

[blois]

I recently added some initialization to Bokeh within Colab that allows it to automatically configure the output frame when necessary- wondering if there are similar hooks which could be used for holoviews?

See https://github.com/googlecolab/colabtools/blob/master/google/colab/_import_hooks/_bokeh.py

[philippjfr]

Yes, I could put together a PR for that. That said that approach is extremely inefficient since it loads bokehJS each time. Can't think of a better solution though.

[blois]

Yes, that is a downside of the isolation between outputs.

We have a security requirement that outputs that were loaded with the notebook be more strictly sandboxed than outputs generated within the current session. Because of this we give each output it's own iframe.

Notebook-level initialization does get a bit tricky when dealing with versioning libraries and edits across multiple sessions. We have found that self-contained outputs can result in more predictability- our goal is that users can always open old notebooks and they will render just as they did originally.

[jbednar]

In the long run, we're already hoping for BokehJS to become more modular, so that not everything needs to be loaded to get anything. Maybe that will help? In the meantime, at least if things work at all, that's a massive improvement over the current situation!

[philippjfr]

Not sure how being more modular would help since to make sure that anything will render you'd still have to load everything.

In the meantime, at least if things work at all, that's a massive improvement over the current situation!

Agreed, but to be clear this would merely automate what you can already do to make it work, which is to load the extension in each cell.

[jbednar]

Sure; it's just that repeating that code in every single cell is so painful and distracting that it doesn't seem like a viable approach. Whereas if it's just slow, then at least the final notebook is still readable if the reloading is hidden.

Whether being modular will help depends on how the modules are chosen, but surely there is some of the JS that isn't needed in every plot!

[kazimuth]

Tip: you can make it a little less repetetive via:

import holoviews as hv
def hvshow(x):
    hv.extension('bokeh')
    return x

Then just call hvshow(e) to show some element.

[korakot]

My hack to avoid hv.extension('bokeh') in every cell

import holoviews as hv

def _render(self, **kw):
  hv.extension('bokeh')
  return hv.Store.render(self)
hv.core.Dimensioned._repr_mimebundle_ = _render

[jlstevens]

@philippjfr I think we might want to consider adding a hook in _repr_mimebundle_ to make it easier for colab users. We could also set the hook I suppose (but I don't want to change the extension code itself), I'm thinking of something like:

import holoviews as hv
hv.config.enable_colab_support = True

[philippjfr]

No huge objection but I'd prefer getting something like this merged into Colab itself. I know that was under discussion at some point.

[philippjfr]

We could also set the hook I suppose (but I don't want to change the extension code itself), I'm thinking of something like

I'd like to use a system like I do in panel where additional pn.extension kwargs are passed through to pn.config.

[jlstevens]

Agreed, makes more sense to do this at the panel level - then holoviews can inherit the fix from there. And yes, getting something merged into Colab would be great!

[philippjfr]

I agree with that but it's actually not what I meant. I meant that I'd like hv.extension to work the same way as pn.extension and pass all config variables through to hv.config.

[teator]

My problem is that DynamicMap doesn't appear in the output of the cell in colab. Does anybody know - is that fixable?

I am also looking forward to having DynamicMap working in Colab.

I found that following code helps to output DynamicMap in output cell in Colab:

from bokeh.plotting import show, output_file
output_file('test.html')
show(hv.render(dm))

Here "dm" is holoviews object.

But the output is just first frame and it is not really dynamic. Can this be improved please?

[philippjfr]

But the output is just first frame and it is not really dynamic. Can this be improved please?

This is not really down to us, Colab is a closed-source, hard fork of the Jupyter project which means it doesn't support any of the standard Comm APIs. We can look into how standard ipywidgets work and maybe add support by wrapping HoloViews output in an ipywidget but there's no guarantee that will actually work.

[blois]

Right now Colab does not support third-party ipywidgets, an explanation of some of the issues is at https://github.com/nteract/nes/blob/master/portable-widgets/readme.md.

I'd really like to see something where we expose Jupyter Comms as an API which can be used from any displayed content, somewhat similar to Jupyter Notebook but without requiring the extension installation step of Jupyter Lab. This way content can be displayed as HTML/JS then use the comms to become more dynamic. Ideally this API would be something which could be implemented in more environments than just Colab.

I would be interested in exposing Comms in an API fully compatible with Jupyter Notebook but as soon as you declare window.jupyter then too much content assumes it can access the entire classic notebook API and breaks.

[philippjfr]

We now have support for comms merged into pyviz_comms, so it would be great if we could also make progress on the hook to load the extension. @blois What's the best approach here? Should I make a PR similar to add something similar to https://github.com/googlecolab/colabtools/blob/master/google/colab/_import_hooks/_bokeh.py?

[blois]

Thanks for all of your work @philippjfr! If you have an example of the changes needed then I can work on getting them into Colab.

Also- we have been working to unblock upgrading the default Bokeh version in Colab for a while- Bokeh requires a Tornado upgrade which in turn requires an ipykernel upgrade, which we need to put through a significant testing/stabilization phase. These upgrades a high priority that we're actively working on, but it's a fairly complex set of changes that are needed.

[23pointsNorth]

Ignore if tangental, but as of today, I still cannot run the code below in colab.

!pip install -U bokeh holoviews
import os
import holoviews as hv # Fails here with `ImportError: cannot import name 'MultiChoice'`
os.environ['HV_DOC_HTML'] = 'true'
hv.extension('bokeh') 

And from what I gathered from the sandboxing, things like sliders are basically not going to happen?

[philippjfr]

This is related to the issues @blois was talking about, i.e. there are issues upgrading to bokeh 2.0 (although it seemed okay when I forced it). You can either downgrade to an older version of Panel or try to force the upgrade to bokeh>2.0. It's being worked on though.

And from what I gathered from the sandboxing, things like sliders are basically not going to happen?

That's what I was talking about in my recent comment, we have merged support for bi-directional comms which means sliders will work as soon as they are enabled for everyone in Colab. @blois can hopefully give you a timeline there.

[blois]

When installing the latest Bokeh you'll want to upgrade ipykernel as well:

!pip install --upgrade bokeh ipykernel

The change to enable comms for everyone in Colab went live this morning.

[korakot]

So, the minimal holoviews code works now.

!pip install -U bokeh ipykernel holoviews
# then restart runtime first, then run
import holoviews as hv
hv.extension('bokeh')
hv.Curve([1, 2, 3])

[korakot]

If you update pyviz_comms to the github, DynamicMap also works.

!pip install -U bokeh ipykernel holoviews git+https://github.com/holoviz/pyviz_comms

Restart then run a minimal DynamicMap

import numpy as np
import holoviews as hv
hv.extension('bokeh')

def sine_curve(freq):
    xvals = [0.1* i for i in range(100)]
    return hv.Curve((xvals, [np.sin(freq*x) for x in xvals]))

dmap = hv.DynamicMap(sine_curve, kdims=['frequency'])
dmap.redim.range(frequency=(0.5,1.25))

Thank you! @blois @philippjfr

[philippjfr]

Perfect, didn't know it was already live. Thanks for checking! I'll tag a pyviz_comms release tomorrow.

[philippjfr]

Also thanks for all your work and all your help here @blois. You rock!

[korakot]

For the original question. I found the answer from looking at holoviews/ipython

It now has an extension that you can load with

%load_ext holoviews.ipython 

Then you don't need to call hv.extension('bokeh') every cell any more. The object can just display itself correctly

hv.Curve([1, 2, 3])

Unfortunately, it didn't work with DynamicMap example above. I might make a mistake somewhere. Here's my colab notebook, in case someone want to help.

[blois]

And thank you @philippjfr - I'm so excited to see this working!

Looking at the sample from @korakot there are still some oddities- I'll dig in soon but any advice or pointers would be helpful.

• The curve rendering after %load_ext holoviews.ipython is inlining a lot of JS (~3MB)- ideally this would be loading from a CDN. Looks like there's an option for that: https://github.com/holoviz/holoviews/blob/master/holoviews/ipython/__init__.py#L86
• The rendering after %load_ext holoviews.ipython doesn't include the controls that appear from hv.extension('bokeh')
• DynamicMap does indeed appear to also interrupt the rendering of other holoviews plots.

[blois]

This morning Colab updated the kernels to include Holoviews 1.13.3 and Bokeh 2.1.1- crossing our fingers that it sticks (required an ipykernel upgrade as well).