Jupyter Notebooks conversion: option to import/export metadata

[chrisjsewell]

I would envisage the format in the .py file being:

#~ key1:
#~   subkey1: value

#%%
#~ key2:
#~   subkey2: value
print('hallo world')

In this format #~ denotes a metadata block, and the blocks are in YAML format.
If there is a block before any cells, this is deemed to be the notebook level metadata.,
and, if its directly below a cell initialiser #%%, it is cell level metadata.

This allows for a more 1:1 mapping between the .py and .ipynb,
and would actually make editing metadata far easier than in the standard web browser environment
(akin to knitr+Sweave feature in R Studio, as @neo-anderson mentions in microsoft/vscode-python#3919 ).

My principal use case for this is with iPyPublish,
which can control (via metadata) how cells and outputs are converted to latex/html/etc
(see here)

Obviously there could be a configuration setting to turn metadata export on/off.

[rchiodo]

This is useful for more than just iPyPublish. Any tool that uses metadata would need this support.

[chrisjsewell]

Heres a freebee: for the importer, this is the jinja template, and associated filter you'd need:

{%- extends 'null.tpl' -%}
{% block header %}
{{ nb.metadata | meta2yaml('#~~ ') }}
{% endblock header %}
{% block codecell %}
#%%
{{ super() }}
{% endblock codecell %}
{% block in_prompt %}{% endblock in_prompt %}
{% block input %}{{ cell.metadata | meta2yaml('#~~ ') }}
{{ cell.source | ipython2python }}
{% endblock input %}
{% block markdowncell scoped %}#%% [markdown]
{{ cell.metadata | meta2yaml('#~~ ') }}
{{ cell.source | comment_lines }}
{% endblock markdowncell %}

import yaml
from nbformat.notebooknode import NotebookNode


def recurse_convert(node):
    # type: (NotebookNode) -> dict
    """convert notebook node to dict"""
    dct = {}
    for key, val in node.items():
        if isinstance(val, NotebookNode):
            dct[key] = recurse_convert(val)
        else:
            dct[key] = val
    return dct


def meta_to_yaml(metadata, comment="#~~ "):
    # type: (NotebookNode, str) -> str
    """convert metadata json to yaml"""
    metadata = recurse_convert(metadata)
    string = yaml.dump(metadata, default_flow_style=False)
    if comment:
        string = "\n".join([comment + s for s in string.splitlines()])
    return string

Since you are calling nbconvert as an executable, I'm not sure how you parse it the filter though?

[rchiodo]

Thanks. Not sure either. Hopefully we can just add python in the tpl file.

[chrisjsewell]

FYI, I've now been pointed towards jupytext,
which more comprehensively handles the round trip conversion of notebook and python script:

>> jupytext --to py:percent notebook.ipynb         # create a notebook.py file in the double percent format
>> jupytext --to notebook notebook.py              # overwrite notebook.ipynb (remove outputs)
>> jupytext --to notebook --update notebook.py     # update notebook.ipynb (preserve outputs)

[greazer]

Thanks for the feedback! However, we don't have plans on adding this functionality at this time.