Error when internationalizing documentation with ipynb files

[iamdalio]

Describe the bug

When attempting to internationalize Sphinx documentation that contains ipynb files using the make -e SPHINXOPTS="-D language='de'" html command, the following error occurs:

Exception occurred:
  File "/usr/local/lib/python3.10/site-packages/nbformat/reader.py", line 26, in parse_json
    raise NotJSONError(message) from e
nbformat.reader.NotJSONError: Notebook does not appear to be JSON: 'Aim dasd 11'

How to Reproduce

1. clone doc files
   git clone https://github.com/hwchase17/langchain.git
2. cd langchain/docs
3. Add configurations to conf.py.
   locale_dirs = ['locale/'] # path is example but recommended.
4. make gettext
5. sphinx-intl update -p _build/gettext -l de
6. Translate po files. **Notice: modify msgstr in ipynb files **

#: ../../ecosystem/aim_tracking.ipynb:10002
msgid "Aim"
msgstr "Aim dasd 11"

7. make -e SPHINXOPTS="-D language='de'" html

Environment Information

Platform:              darwin; (macOS-13.1-x86_64-i386-64bit)
Python version:        3.10.8 (main, Oct 13 2022, 10:17:43) [Clang 14.0.0 (clang-1400.0.29.102)])
Python implementation: CPython
Sphinx version:        6.2.1
Docutils version:      0.19
Jinja2 version:        3.1.2
Pygments version:      2.14.0

Sphinx extensions

extensions = [
    "sphinx.ext.autodoc",
    "sphinx.ext.autodoc.typehints",
    "sphinx.ext.autosummary",
    "sphinx.ext.napoleon",
    "sphinx.ext.viewcode",
    "sphinxcontrib.autodoc_pydantic",
    "myst_nb",
    "sphinx_copybutton",
    "sphinx_panels",
    "IPython.sphinxext.ipython_console_highlighting",
]

Additional context

No response

[iamdalio]

I tried to troubleshoot the issue and found that

1. building the HTML without internationalization works fine. Also,
2. converting the IPYNB file to Markdown and then internationalizing it works fine.
3. changing myst_nb extension to nbsphinx , the following error occurs:
   docs/ecosystem/aim_tracking.ipynb:7: ERROR: Unexpected indentation.

[BillHuang2001]

Same problem here.
It seems that the i18n code is eagerly parsing each piece of message, and the notebook parser (myst_nb or nbsphinx) is called with only that string, not the whole notebook.
A potential workaround is to wrap every translated string with jupyter notebook's json format {"cell": [{"source": [... "translated" ...]}]}. It will compile, but I haven't check if the final result is valid.

Traceback

Traceback (most recent call last):
  File "/----/lib/python3.11/site-packages/sphinx/cmd/build.py", line 298, in build_main
    app.build(args.force_all, args.filenames)
  File "/----/lib/python3.11/site-packages/sphinx/application.py", line 355, in build
    self.builder.build_update()
  File "/----/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 293, in build_update
    self.build(to_build,
  File "/----/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 313, in build
    updated_docnames = set(self.read())
                           ^^^^^^^^^^^
  File "/----/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 420, in read
    self._read_serial(docnames)
  File "/----/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 441, in _read_serial
    self.read_doc(docname)
  File "/----/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 498, in read_doc
    publisher.publish()
  File "/----/lib/python3.11/site-packages/docutils/core.py", line 236, in publish
    self.apply_transforms()
  File "/----/lib/python3.11/site-packages/docutils/core.py", line 216, in apply_transforms
    self.document.transformer.apply_transforms()
  File "/----/lib/python3.11/site-packages/sphinx/transforms/__init__.py", line 83, in apply_transforms
    super().apply_transforms()
  File "/----/lib/python3.11/site-packages/docutils/transforms/__init__.py", line 182, in apply_transforms
    transform.apply(**kwargs)
  File "/----/lib/python3.11/site-packages/sphinx/transforms/i18n.py", line 397, in apply
    patch = publish_msgstr(self.app, msgstr, source,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/----/lib/python3.11/site-packages/sphinx/transforms/i18n.py", line 73, in publish_msgstr
    doc = reader.read(
          ^^^^^^^^^^^^
  File "/----/lib/python3.11/site-packages/docutils/readers/__init__.py", line 70, in read
    self.parse()
  File "/----/lib/python3.11/site-packages/docutils/readers/__init__.py", line 76, in parse
    self.parser.parse(self.input, document)
  File "/----/lib/python3.11/site-packages/myst_nb/sphinx_.py", line 89, in parse
    notebook = nb_reader.read(inputstring)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/----/lib/python3.11/site-packages/myst_nb/core/read.py", line 38, in standard_nb_read
    return nbf.reads(text, as_version=NOTEBOOK_VERSION)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/----/lib/python3.11/site-packages/nbformat/__init__.py", line 89, in reads
    nb = reader.reads(s, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/----/lib/python3.11/site-packages/nbformat/reader.py", line 76, in reads
    nb_dict = parse_json(s, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/----/lib/python3.11/site-packages/nbformat/reader.py", line 26, in parse_json
    raise NotJSONError(message) from e
nbformat.reader.NotJSONError: Notebook does not appear to be JSON: 'some translated text'

[mgeier]

I don't think this is a problem of Sphinx (at least not immediately), so this should be reported to the affected extensions.

For nbsphinx, this has already be reported at spatialaudio/nbsphinx#727, but due to lack of response there hasn't been any real progress.

[picnixz]

I'll close the issue since it's not directly related to Sphinx core. Please report the issue to nbsphinx (should be the one responsible for nbformat). If they tell you that the problem is here, feel free to reopen the issue (or create a new one if it is locked).

[BillHuang2001]

Thanks for the information.
In case anyone is looking for the workaround I was talking about, I would like to leave my code in here

workaround

import polib
import json

po = polib.pofile("<path to the .po file>")


def is_from_notebook(occurrences):
    for filename, _linenum in occurrences:
        if filename.endswith(".ipynb"):
            return True

    return False


for entry in po:
    if is_from_notebook(entry.occurrences) and entry.msgstr != "":
        wrapped = {
            "cells": [
                {
                    "cell_type": "markdown",
                    "metadata": {},
                    "source": [entry.msgstr],
                },
            ],
            "metadata": {
                "kernelspec": {
                    "display_name": "venv",
                    "language": "python",
                    "name": "python3",
                },
                "language_info": {
                    "codemirror_mode": {"name": "ipython", "version": 3},
                    "file_extension": ".py",
                    "mimetype": "text/x-python",
                    "name": "python",
                    "nbconvert_exporter": "python",
                    "pygments_lexer": "ipython3",
                    "version": "3.10.12",
                },
            },
            "nbformat": 4,
            "nbformat_minor": 2,
        }
        entry.msgstr = json.dumps(wrapped)

po.save("<path to the .po file>")

At least for me, this code will resolve the error and the rendered text appears to be correct as well.