Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jinja2.exceptions.TemplateNotFound: toc2 #1533

Open
elgalu opened this issue Oct 9, 2020 · 13 comments
Open

jinja2.exceptions.TemplateNotFound: toc2 #1533

elgalu opened this issue Oct 9, 2020 · 13 comments

Comments

@elgalu
Copy link

elgalu commented Oct 9, 2020

jinja2.exceptions.TemplateNotFound: toc2 failure after upgrading nbconvert >= 6

How to reproduce

pip install nbformat "nbconvert>=6"

jupyter nbconvert --to html_toc "/opt/conda/lib/python3.7/site-packages/nbformat/tests/test2.ipynb"

Error

[NbConvertApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `NbConvertApp`.
[NbConvertApp] Converting notebook /opt/conda/lib/python3.7/site-packages/nbformat/tests/test2.ipynb to html_toc
Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.7/site-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 350, in start
    self.convert_notebooks()
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/html.py", line 119, in from_notebook_node
    return super().from_notebook_node(nb, resources, **kw)
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 384, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 148, in template
    self._template_cached = self._load_template()
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 355, in _load_template
    return self.environment.get_template(template_file)
  File "/opt/conda/lib/python3.7/site-packages/jinja2/environment.py", line 830, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/opt/conda/lib/python3.7/site-packages/jinja2/environment.py", line 804, in _load_template
    template = self.loader.load(self, name, globals)
  File "/opt/conda/lib/python3.7/site-packages/jinja2/loaders.py", line 408, in load
    raise TemplateNotFound(name)
jinja2.exceptions.TemplateNotFound: toc2

Workaround

pip install "nbconvert<6"
deflaux added a commit to all-of-us/workbench-snippets that referenced this issue Jan 28, 2021
Smoke test changes include:
* recent SQL from dataset builder for the dataset snippets smoke test setup
* some naming consistency updates within the dataset snippets smoke test setup configuration
* some conditional logic so that we do not attempt to render ipywidgets in the context of a smoke test
* Also note that the newer version of `nbconvert` is incompatible with Jupyter extension `toc2`.
  * This causes the [HTML snapshot](https://github.com/all-of-us/workbench-snippets/tree/master/py#create-and-view-html-snapshots-of-notebooks) functionality to fail as the export format is `html_toc` which is provided by `toc2`.
  * See ipython-contrib/jupyter_contrib_nbextensions#1533
  * For now we work around this by changing the export format to `html`.
@daliagachc
Copy link

is it any roadmap to update to nbconvert>6

@pawlodkowski
Copy link

pawlodkowski commented Dec 20, 2021

seconded. would love if I could get the table of contents html export to work in nbconvert > 6. spent all day working on trying to create a custom exporter myself, but wasn't able to manage.

@leodrivera
Copy link

I've been using this workaround reported by @elgalu, as there are still problems with toc export, even when using the version 5.3, that hasn't been officially published yet.

I've written about the issues I got in the other topic related with TOC #1568

@deflaux
Is there a roadmap for fixing the issue?

@deflaux
Copy link

deflaux commented Jan 18, 2022

@leodrivera I wouldn't know. I have not contributed to this repository. But whenever it is fixed, I look forward to using it again!! The TOC is so nice to have when the Jupyter notebooks exported to HTML are long.

@leodrivera
Copy link

Indeed. As I usually write long notebooks, it makes so much easier for reading it.

@deflaux
This extension is so much useful and if we could get working, would be something very useful for the community.

I've looked at the last commits and I am pinging them, so maybe we can try to solve this together:
@jfbercher
@cailiang9
@aduriseti
@juhasch
@zthxxx

@cailiang9
Copy link
Contributor

maybe this patch to nbconvert can work:

*** a/nbconvert/exporters/templateexporter.py	2020-11-07 12:54:46.000000000 +0800
--- b/nbconvert/exporters/templateexporter.py	2020-11-08 01:22:31.634876123 +0800
***************
*** 531,555 ****
--- 531,562 ----
              # we include root_dir for when we want to be very explicit, e.g.
              # {% extends 'nbconvert/templates/classic/base.html' %}
              paths.append(root_dir)
              # we include base_dir for when we want to be explicit, but less than root_dir, e.g.
              # {% extends 'classic/base.html' %}
              base_dir = os.path.join(root_dir, 'nbconvert', 'templates')
              paths.append(base_dir)
  
+             compatibility_dir = os.path.join(root_dir, 'jupyter_contrib_nbextensions', 'templates')
+             paths.append(compatibility_dir)
+             compatibility_dir = os.path.join(root_dir, 'nbconvert', 'templates', 'base')
+             paths.append(compatibility_dir)
+             compatibility_dir = os.path.join(root_dir, 'nbconvert', 'templates', 'classic')
+             paths.append(compatibility_dir)
              compatibility_dir = os.path.join(root_dir, 'nbconvert', 'templates', 'compatibility')
              paths.append(compatibility_dir)
  
          additional_paths = self.template_data_paths
          for path in additional_paths:
              try:
                  ensure_dir_exists(path, mode=0o700)
              except OSError:
                  pass
+         #print("self.extra_template_basedirs:", self.extra_template_basedirs, self.extra_template_paths + additional_paths + paths)
  
          return self.extra_template_paths + additional_paths + paths
  
      @classmethod
      def get_compatibility_base_template_conf(cls, name):
          # Hard-coded base template confs to use for backwards compatibility for 5.x-only templates
          if name == 'display_priority':
              return dict(base_template='base')

@cailiang9
Copy link
Contributor

according to https://nbconvert.readthedocs.io/en/latest/external_exporters.html#registering-a-custom-exporter-as-an-entry-point, you can also try:
jupyter nbconvert --to jupyter_contrib_nbextensions.nbconvert_support.toc2.TocExporter notebook.ipynb

@leodrivera
Copy link

leodrivera commented Jan 19, 2022

@cailiang9
Thanks for the answer

Unfortunately, it did not solve for me. However, I could solve part of the problem by changing the files hosted in https://gitcdn.xyz/ (and substituting the path here) to the same files hosted in my personal github (inside github.io) and, after that, the js could be effectively loaded and the page appeared.
It seems that the gitcdn site is very slow and unstable. It would be very nice if another repo was created, with the github pages options enabled, so that these files could be hosted into it.

But I still have those two issues:

@Sieboldianus
Copy link

Sieboldianus commented Jul 26, 2022

I got toc2 rendered correctly in a notebook html in nbconvert 6.5.0 and jupyterlab 3.4.4. The key was:

  • add a custom template and use custom URLs for toc2 sources (CDN) main.css and toc2.js
    <link rel="stylesheet" type="text/css" href="https://my.cdn.org/toc2/main.css">
    <script src="https://my.cdn.org/toc2/toc2.js"></script>
  • add a conf.json referencing the classic template (instead of full.tpl)
{
  "base_template": "classic",
  "mimetypes": {
    "text/html": true
  },
  "preprocessors": {
    "100-pygments": {
        "type": "nbconvert.preprocessors.CSSHTMLHeaderPreprocessor",
        "enabled": true
    },
    "500-reveal": {
      "type": "nbconvert.exporters.slides._RevealMetadataPreprocessor",
      "enabled": true
    }
  }
}
  • in the custom template, instead of full.tpl, extend from classic/index.html.j2
{%- extends 'classic/index.html.j2' -%}
  • add a block to your template, reversing the nbconvert order of requirejs and jquery - see my report here.
{%- block html_head_js -%}
{%- block html_head_js_requirejs -%}
<script src="{{ resources.require_js_url }}"></script>
{%- endblock html_head_js_requirejs -%}
{%- block html_head_js_jquery -%}
<script src="{{ resources.jquery_url }}"></script>
{%- endblock html_head_js_jquery -%}
{%- endblock html_head_js -%}

Here is my full working template, you only need to replace URLs for main.css and toc2.js - I don't want mine to be used elsewhere.

nbconvert.tpl
{%- extends 'classic/index.html.j2' -%}

{# toc2 template for nbconvert >6.0
- created from files in:
    https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/templates
    https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/toc2
    
    https://github.com/ipython-contrib/jupyter_contrib_nbextensions/blob/master/src/jupyter_contrib_nbextensions/templates/toc2.tpl
    https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/toc2
#}
    
{% block output_group -%}
{%- if cell.metadata.hide_output -%}
{%- else -%}
    {{ super() }}
{%- endif -%}
{% endblock output_group %}

{% block output_area_prompt %}
{%- if cell.metadata.hide_input or nb.metadata.hide_input -%}
    <div class="prompt"> </div>
{%- else -%}
    {{ super() }}
{%- endif -%}
{% endblock output_area_prompt %}

{%- block html_head_js -%}
{%- block html_head_js_requirejs -%}
<script src="{{ resources.require_js_url }}"></script>
{%- endblock html_head_js_requirejs -%}
{%- block html_head_js_jquery -%}
<script src="{{ resources.jquery_url }}"></script>
{%- endblock html_head_js_jquery -%}
{%- endblock html_head_js -%}

{%- block html_head -%}

{{ super() }}

{% for css in resources.inliner.css -%}
    <style type="text/css">
    {{ css }}
    </style>
{% endfor %}

{% for js in resources.inliner.js -%}
    <script type="text/javascript">
    {{ js }}
    </script>
{% endfor %}

    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.12.1/jquery-ui.css">
    
    <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.9.1/jquery-ui.min.js"></script>
    
    <link rel="stylesheet" type="text/css" href="https://replace.with.cdn/toc2/main.css">
    
    <link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
    
    <script src="https://replace.with.cdn/toc2/toc2.js"></script>
    
    <script>
    $( document ).ready(function(){
            var cfg = {{ nb.get('metadata', {}).get('toc', {})|tojson|safe }};
            cfg.navigate_menu=false;
            // fire the main function with these parameters
            require(['nbextensions/toc2/toc2'], function (toc2) {
                toc2.table_of_contents(cfg);
            });
    });
    </script>
    
{%- endblock html_head -%}

Then export your notebook with:

!jupyter nbconvert --to html_toc \
    --output-dir=./html ./test.ipynb \
    --template=./nbconvert.tpl \
    --ExtractOutputPreprocessor.enabled=False  >&- 2>&- # create single output file and suppress output

(from a notebook cell)

screenshot

image

@palbarede
Copy link

palbarede commented Feb 14, 2023

% jupyter notebook --version
6.5.2
% pip list|grep nbconvert
nbconvert                         7.2.9

I still have the problem. I am trying your solution.

In my installation, I find toc2.js and main.js in

/home/alba/.conda/envs/jupyter/lib/python3.10/site-packages/jupyter_contrib_nbextensions/nbextensions/toc2/,

which I replace exactly for 'https://replace.with.cdn/toc2/' in your nbconvert.tpl. (Actually this does not change anything in the following.)

I find
/home/alba/.conda/envs/jupyter/share/jupyter/nbconvert/templates/compatibility/full.tpl,

% cat ~/.conda/envs/jupyter/share/jupyter/nbconvert/templates/compatibility/full.tpl {{ resources.deprecated("This template is deprecated, please use classic/index.html.j2") }} {%- extends 'index.html.j2' -%}

I think it is almost what you have done.

I guess that your text beginning with

{
  "base_template": "classic",

should go in some file named conf.json.

I found

% cat /home/alba/.conda/envs/jupyter/share/jupyter/nbconvert/templates/classic/conf.json
{
  "base_template": "base",
  "mimetypes": {
    "text/html": true
  },
  "preprocessors": {
    "100-pygments": {
      "type": "nbconvert.preprocessors.CSSHTMLHeaderPreprocessor",
      "enabled": true,
      "style": "default"
    }
  }
}

I replace it by your conf.json, including "base_template": "classic".

I also change the first line of /home/alba/.conda/envs/jupyter/share/jupyter/nbconvert/templates/classic/ so that

% head index.html.j2      
{%- extends 'classic/index.html.j2' -%}

Then the ouput of your program is:

File "/home/alba/.conda/envs/jupyter/share/jupyter/nbconvert/templates/classic/index.html.j2", line 2, in top-level template code
    {% from 'mathjax.html.j2' import mathjax %}
jinja2.exceptions.TemplateNotFound: mathjax.html.j2

The problem https://github.com/jupyter/nbconvert/issues/1394 has been solved.

Then

% pwd                       
/home/alba/.conda/envs/jupyter/share/jupyter/nbconvert/templates/classic
% cp ../base/mathjax.html.j2 .

and following the same principle as before add classic/ before mathjax.html.j2 in index.html.j1.

Similar correction needed for jupyter_widgets.html.j2.

Next error:

  File "/home/alba/.conda/envs/jupyter/lib/python3.10/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison

abc.py is the abstract class Python module.

I put print(cls) at line 119 of abc.py.

File "/home/alba/.conda/envs/jupyter/share/jupyter/nbconvert/templates/classic/index.html.j2", line 3, in top-level template code
    {% from 'classic/jupyter_widgets.html.j2' import jupyter_widgets %}
  [Previous line repeated 973 more times]
  File "/home/alba/.conda/envs/jupyter/share/jupyter/nbconvert/templates/classic/index.html.j2", line 2, in top-level template code
    {% from 'classic/mathjax.html.j2' import mathjax %}
  File "/home/alba/.conda/envs/jupyter/lib/python3.10/_collections_abc.py", line 666, in __rsub__
    if not isinstance(other, Set):
  File "/home/alba/.conda/envs/jupyter/lib/python3.10/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison

@lukelbd
Copy link

lukelbd commented Sep 17, 2023

Spent a lot of time trying to get table of contents to work and just could not get it done. @Sieboldianus I added your nbconvert.tpl template + conf.json file and successfully used it with --to html_toc --template ./nbconvert.tpl, but I just get None printed at the top of the page. Also made sure to enable table of contents sidebar for the notebook in question by following this stackoverflow answer, and manually enabling in jupyter notebook interface.

This is what nbextensions configurator tab shows, and this is the notebook in question:

Screenshot 2023-09-17 at 18 27 54

Might be some weird install issue with toc2 but not sure. Not sure where to go from here, giving up for now. For reference here is my current environment (also using jinja2 3.0.0):

$ pip list | grep -E 'nbconvert|jinja2|nbextension|jupyter
nbconvert                         7.8.0
jupyter_client                    7.4.9
jupyter-console                   6.6.3
jupyter-contrib-core              0.4.0
jupyter-contrib-nbextensions      0.7.0
jupyter_core                      5.3.0
jupyter-events                    0.7.0
jupyter-highlight-selected-word   0.2.0
jupyter-latex-envs                1.4.6
jupyter-lsp                       2.2.0
jupyter-nbextensions-configurator 0.6.1
jupyter_server                    2.7.3
jupyter_server_fileid             0.9.0
jupyter_server_terminals          0.4.4
jupyter_server_ydoc               0.8.0
jupyter-ydoc                      0.2.4
jupyterlab                        3.6.5
jupyterlab-pygments               0.2.2
jupyterlab_server                 2.25.0

@Sieboldianus
Copy link

Sieboldianus commented Sep 18, 2023

Yes, this gets more and more complicated. My approach worked until last week, when jupyter_contrib_nbextensions failed to install with conda (see #1650). I could work around this by installation with pip. However, then I got a ModuleNotFoundError: No module named 'notebook.base' error. I could work around this by pinning notebook<7.0.0.

These are my current versions:

notebook 6.5.5
jupyter_server 2.7.3
jupyterlab 4.0.6
nbconvert 7.8.0

Here are some links to get you started:

Some hints:

  • I don't use jupyter_nbextensions_configurator anymore since this seems not compatible anymore with the latest Jupyter lab
  • I don't use the toc extension to display toc inside the notebook; I only need it to render/add a toc to nbconvert-generated html files

@zertrin
Copy link

zertrin commented Sep 23, 2023

I struggled a lot combining all the knowledge in this thread for my case, but finally stumbled a working solution for my use case (using nbconvert<6).

I'm working in an older conda environment with an older version of nbconvert 5.6.1 (for scientific notebooks developed some years ago), and I managed to circumvent both the missing toc2 template issue and also the current problem whereby raw.gitcdn.xyz is now down.

Here are my steps (hoping this can help someone):

  • copy toc2.tpl file from $HOME/mambaforge/envs/my-env-name-redacted/lib/python3.7/site-packages/jupyter_contrib_nbextensions/templates/toc2.tpl to a folder of my choice (let's say $HOME/custom-toc2/)
  • modified the URLs in $HOME/custom-toc2/toc2.tpl
    • by changing https://min.gitcdn.xyz/cdn/ipython-contrib/jupyter_contrib_nbextensions/master/src/...
    • to https://cdn.jsdelivr.net/gh/ipython-contrib/jupyter_contrib_nbextensions@0.5.1/src/...
    • for both the main.css and toc2.js imports
Resulting $HOME/custom-toc2/toc2.tpl (click to expand)
{%- extends 'nbextensions.tpl' -%}


{%- block header -%}
{{ super() }}

 <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.12.1/jquery-ui.css">

<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.9.1/jquery-ui.min.js"></script>

<link rel="stylesheet" type="text/css" href="https://cdn.jsdelivr.net/gh/ipython-contrib/jupyter_contrib_nbextensions@0.5.1/src/jupyter_contrib_nbextensions/nbextensions/toc2/main.css">

<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">

<script src="https://cdn.jsdelivr.net/gh/ipython-contrib/jupyter_contrib_nbextensions@0.5.1/src/jupyter_contrib_nbextensions/nbextensions/toc2/toc2.js"></script>

<script>
$( document ).ready(function(){

            var cfg = {{ nb.get('metadata', {}).get('toc', {})|tojson|safe }};
            cfg.navigate_menu=false;
            // fire the main function with these parameters
            require(['nbextensions/toc2/toc2'], function (toc2) {
                toc2.table_of_contents(cfg);
            });
    });
</script>

{%- endblock header -%}

Then finally, using the following command to convert my notebooks:

jupyter nbconvert --to html_toc --template="$HOME/custom-toc2/toc2.tpl" my-notebook.ipynb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants