Update nbconvert to work with Altair/Vega-Lite/Vega output

[Znafon]

Hi,

when exporting q notebook to HTML from the command line, altair does not export figures and they are not embedded in the produced document.

Exporting the notebook from the jupyter web interface works correctly and exports every figures.

Thanks,
Rémi

[jakevdp]

This sounds like an issue in http://github.com/vega/ipyvega, which is soon going to be replaced by https://github.com/altair-viz/jupyter_vega. @gnestor, do you know whether this will still be an issue in jupyter_vega?

[gnestor]

It sounds like an issue with the mime type of the figures. In jupyter_vega, we output both a vega mime type that renders the dynamic plot and a state image/png output that can be rendered in nbviewer and when a notebook is converted to HTML.

See #216 for status on built-in support Vega/Vega-lite in JupyterLab. Once this is merged, we will do the same for classic Jupyter Notebook.

[ellisonbg]

We have plans to do a full overhaul of nbconvert's HTML oputput eventually to support arbitrary MIME output based on the new JupyterLab components. But that may still be a ways off as JupyterLab is still pre-1.0. @mpacer @blink1073 @sccolbert @jasongrout

[gnestor]

@ellisonbg I like this plan! Until then, should renderer extensions still plan on providing an image/png or text/html representation along with the custom MIME representation to support rendering in nbviewer/on Github?

[ellisonbg]

Yes, I think so!

…

On Wed, Sep 27, 2017 at 1:24 PM, Grant Nestor ***@***.***> wrote: @ellisonbg <https://github.com/ellisonbg> I like this plan! Until then, should renderer extensions still plan on providing an image/png or text/html representation along with the custom MIME representation to support rendering in nbviewer/on Github? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#329 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0E-GxrmbEI4xz5_RdJHrMR-3n2Vxks5smq8agaJpZM4NfaM3> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[tanyaschlusser]

@gnestor I have a workaround that just uses Altair to generate charts and runs vegaEmbed as javascript directly in an HTML() command. (Shameless plug: here's my working Kaggle notebook with it.)

First set up to use the javascript libraries

Run this in a separate cell because the last thing has to be the HTML() call.

import json  # need it for json.dumps
import altair as alt
from altair.vega import v3

# Create the correct URLs for require.js to find the Javascript libraries
vega_url = 'https://cdn.jsdelivr.net/npm/vega@' + v3.SCHEMA_VERSION
vega_lib_url = 'https://cdn.jsdelivr.net/npm/vega-lib'
vega_lite_url = 'https://cdn.jsdelivr.net/npm/vega-lite@' + alt.SCHEMA_VERSION
vega_embed_url = 'https://cdn.jsdelivr.net/npm/vega-embed@3'
noext = "?noext"

paths = {
    'vega': vega_url + noext,
    'vega-lib': vega_lib_url + noext,
    'vega-lite': vega_lite_url + noext,
    'vega-embed': vega_embed_url + noext
}

workaround = """
requirejs.config({{
    baseUrl: 'https://cdn.jsdelivr.net/npm/',
    paths: {paths}
}});
"""

HTML("".join((
    "<script>",
    workaround.format(paths=json.dumps(paths)),
    "</script>",
    "This code block sets up embedded rendering in HTML."
)))

Next make a helper function to do the rendering

It is called render() and takes a dictionary or an alt.Chart.

# Define the function for rendering
def add_autoincrement(render_func):
    # Keep track of unique <div/> IDs
    cache = {}
    def wrapped(chart, id="vega-chart", autoincrement=True):
        """Render an altair chart directly via javascript.
        
        This is a workaround for functioning export to HTML.
        (It probably messes up other ways to export.) It will
        cache and autoincrement the ID suffixed with a
        number (e.g. vega-chart-1) so you don't have to deal
        with that.
        """
        if autoincrement:
            if id in cache:
                counter = 1 + cache[id]
                cache[id] = counter
            else:
                cache[id] = 0
            actual_id = id if cache[id] == 0 else id + '-' + str(cache[id])
        else:
            if id not in cache:
                cache[id] = 0
            actual_id = id
        return render_func(chart, id=actual_id)
    # Cache will stay defined and keep track of the unique div Ids
    return wrapped


@add_autoincrement
def render(chart, id="vega-chart"):
    # This below is the javascript to make the chart directly using vegaEmbed
    chart_str = """
    <div id="{id}"></div><script>
    require(["vega-embed"], function(vegaEmbed) {{
        const spec = {chart};     
        vegaEmbed("#{id}", spec, {{defaultStyle: true}}).catch(console.warn);
    }});
    </script>
    """
    return HTML(
        chart_str.format(
            id=id,
            chart=json.dumps(chart) if isinstance(chart, dict) else chart.to_json(indent=None)
        )
    )

Then you can use the render function like this

import pandas as pd

df = pd.DataFrame(dict(a=tuple('ABCDEFGHI'), b=(28, 55, 43, 90, 81, 53, 19, 87, 52)))
render(
    alt.Chart(df, width=360).mark_bar().encode(x='a:O', y='b:Q', tooltip='b')
)

Hope it is useful to someone.
Awesome library, folks!

[Alcampopiano]

@tanyaschlusser This really saved my day. Thank you.

[Hugovdberg]

An alternative is to customise the jinja2 template that nbconvert uses. I use this template: https://gist.github.com/Hugovdberg/96f0645ad13717d282026222eb5c32bf which seems to be sufficient to get plots to be rendered interactively.

In short it adds the vega JavaScript libraries to the header, and wraps the specs for each visualisation in a div followed by a vegaEmbed command. It currently assumes only a single output per cell (uses the execution_number for the numbering), so it might fail if multiple plots have the same execution_number.

jupyter nbconvert --to html notebook.ipynb --template=altair_interactive.tpl output.html

[Hugovdberg]

Also, this gist ensures unique numbers, although it is slightly hacky to get around the scoped context. Assignment doesn't work, but calling a function on a mutable object works just fine.

[hassenmorad]

@tanyaschlusser I'm eager to implement your workaround but received a NameError when running the HTML() command: name 'HTML' is not defined. I tried: from html import HTML but received an ImportError: cannot import name 'HTML'. Any suggestions on how to fix this?

[Alcampopiano]

@hassenmorad I think you'll want:
from IPython.display import HTML

[hassenmorad]

@Alcampopiano That was it. Thanks!

[hassenmorad]

For some reason I'm just getting a blank output (even w/ the exact example @tanyaschlusser provided). Not sure what I'm missing. Any suggestions?

[Alcampopiano]

@hassenmorad Yes this is was happens in the notebook. Save it as HTML and you'll see the charts and they'll have interactivity.

[hassenmorad]

Hmm, not sure what I'm doing wrong. I saved the notebook as html via nbviewer but the html file just displays the static notebook. And when I upload it to github it displays it as html code.

[knanne]

Here is a gist using simple extension of the nbconvert template.

Use:

1. save the below code as nbconvert_template_altair.tpl
2. run jupyter nbconvert --to html --template nbconvert_template_altair.tpl <YOUR_NOTEBOOK.ipynb>

{% extends "full.tpl" %}

{% set altair = {'vis_number': 1} %}

{% block header %}
  <script src="https://cdn.jsdelivr.net/npm/vega@3"></script>
  <script src="https://cdn.jsdelivr.net/npm/vega@5"></script>
  <script src="https://cdn.jsdelivr.net/npm/vega-lite@2"></script>
  <script src="https://cdn.jsdelivr.net/npm/vega-lite@3"></script>
  <script src="https://cdn.jsdelivr.net/npm/vega-embed@3"></script>
  {{super()}}
{% endblock header %}

{% block data_png scoped %}
    {% if 'application/vnd.vegalite.v1+json' in output.data %}
    {% elif 'application/vnd.vegalite.v2+json' in output.data %}
    {% elif 'application/vnd.vegalite.v3+json' in output.data %}
    {% elif 'application/vnd.vega.v2+json' in output.data %}
    {% elif 'application/vnd.vega.v3+json' in output.data %}
    {% else %}
        {{super()}}
    {% endif %}
{% endblock data_png %}

{% block data_text scoped %}
    {% if 'application/vnd.vegalite.v1+json' in output.data %}
    {% elif 'application/vnd.vegalite.v2+json' in output.data %}
    {% elif 'application/vnd.vegalite.v3+json' in output.data %}
    {% elif 'application/vnd.vega.v2+json' in output.data %}
    {% elif 'application/vnd.vega.v3+json' in output.data %}
    {% else %}
        {{super()}}
    {% endif %}
{% endblock data_text %}

{% block data_priority scoped %}
   {% for mimetype in (
        'application/vnd.vegalite.v1+json',
        'application/vnd.vegalite.v2+json',
        'application/vnd.vegalite.v3+json',
        'application/vnd.vega.v2+json',
        'application/vnd.vega.v3+json')
    %}
        {% if mimetype in output.data %}
            {% if altair.update({'vis_number': altair.vis_number+1}) %}{% endif %}
            <div id="vis{{cell['execution_count']}}_{{ altair.vis_number }}"></div>
            <script type="text/javascript">
                var spec = {{ output.data[mimetype] | replace("None","null") | replace("True","true") | replace("False","false") }};
                var opt = {"renderer": "canvas", "actions": false};
                vegaEmbed("#vis{{cell['execution_count']}}_{{ altair.vis_number }}", spec, opt);
            </script>
        {% elif loop.index == 1 %}
            {{super()}}
        {% endif %}
    {% endfor %}
{% endblock data_priority %}

Live Example: https://knanne.github.io/notebooks/visualize_strava_data_in_python.html#Plots

Credit:

• based off original posts (Update nbconvert to work with Altair/Vega-Lite/Vega output #329 (comment)) by @Hugovdberg provided above
• improved with suggestions by @octavifs

Requirements:

• Tested on altair 3.0.1 - 3.2.0
• Tested on jupyterlab 0.35.1 - 1.1.0

Explanations:

• depending on you Altair version, different Vega versions are required and MIME types are produced. All are listed and checked for in the Gist above. This may be overkill.
• static image output in data/png key is excluded in favor of interactive chart for cells with vega output. This is done in the {% block data_png scoped %} snippet
• static text output in data/text key is excluded in favor of interactive chart for cells with vega output. This is done in the {% block data_text scoped %} snippet
• Python None values are not recognized in the Javascript Vega Spec. The workaround is to replace all using replace("None","null")
• Python Boolean values are not recognized in the Javascript Vega Spec. The workaround is to replace all using replace("True","true") | replace("False","false")
• the gist assumes a single Vega Spec per Jupyter cell output. For multiple charts, use compound charts in Altair

[octavifs]

Hi @knanne,
I was having some issues running this template with the new 3.x version of Altair due to the new mimetype (application/vnd.vegalite.v3+json) and different rendering strategy that jupyter lab uses (it adds a 'text_plain' helper text (which should not be rendered) in addition to the vegalite output, but no png.

I also had to tweak around the var spec replace method, as there were some None values that were not being properly transformed to null.

I've published an updated gist with those changes. Feel free to merge them into yours.

I've also created a post on it, plus some examples showing how it all fits together.

[Roald87]

Thanks @octavifs your solution worked for me in the end. Unfortunately @tanyaschlusser and @knanne didn't work for me.

[knanne]

#329 (comment) has been updated with improvements by @octavifs. Working on latest version altair==3.2.0 and jupyterlab==1.1.0

[Juan-132]

Thanks @knanne -- your template works great

@ellisonbg NBConvert currently exports VegaLite plots as static PNG images.
Using the altair chart save API however produces an HTML page with interactive VegaLite plots.

Is this something that can be easily realized with NBConvert as well? (but for the whole notebook)
Or any workarounds we could explore?

[jakevdp]

As of Altair 4.0, the default renderer is HTML-based and should work with nbconvert with interactive charts without any special setup.

The notebook and jupyterlab renderers use frontend-specific mechanisms to render charts, and will not work with nbconvert unless you do some extra setup or templating to handle that type of output. Note that this is not a problem with Altair: this is a general feature of mimebundle-based rich display in the Jupyter ecosystem.

I'm going to close this issue; I think the core questions are addressed as of 4.0.

[jetilton]

Not sure what I am doing wrong, but I cannot export an interactive chart. I have tried to do it in jupyterlab as well as notebook, due to what @jakevdp mentioned above, without any use of templates. When I try to use the provided template or the updated one @octavifs provided, I get a template not found error.

All the templates are within the venv\share\jupyter\nbconvert\templates\ directory, but are split up into sub directories such as base, lab, compatibility, etc. When I pull the failing template out and place it into the main template directory it works, but then fails on the next needed template. Is there a specific location I need to place the above provided template and an expected place to execute the nbconvert command?

I am running the below command:
jupyter nbconvert --to html --template nbconvert_template_altair.tpl Untitled1.ipynb
altair==4.1.0

Thanks

[Juan-132]

How about: jupyter nbconvert Untitled1.ipynb

Please also check that you have the latest versions of Jupyterlab and Nbconvert

[JILPulvino]

An FYI, the altair visuals were not rendering for me when converting to html with nbconvert as well, but found the issue was simply a button at the top of the display to trust HTML that enabled them to render.

jupyter core : 4.7.1 jupyter-notebook : 6.1.4 qtconsole : 5.1.1 ipython : 7.24.1 ipykernel : 5.5.5 jupyter client : 6.1.12 jupyter lab : 3.0.16 nbconvert : 6.1.0 ipywidgets : 7.5.1 nbformat : 5.1.3 traitlets : 5.0.5
and
altair: 4.1.0