[BUG] More than 1 Block Quote in Text causes IndexError when converting to ipynb

[mmcky]

Describe the bug

If there are more than one block quotes in a myst.md file then jupytext (via markdown-it-py) causes

 File "anaconda3/envs/qe-lectures/lib/python3.8/site-packages/markdown_it/rules_block/state_block.py", line 134, in skipEmptyLines
    if (self.bMarks[from_pos] + self.tShift[from_pos]) < self.eMarks[from_pos]:
IndexError: list index out of range

To Reproduce

A minimal Example

---
jupytext:
  text_representation:
    extension: .md
    format_name: myst
kernelspec:
  display_name: Python 3
  language: python
  name: python3
---

# Simple Example

> A Block Quote

some text

> Another Block Quote

and then run jupytext f<ile.md> --to ipynb

Expected behavior

Notebook output with a single cell containing markdown and two block-quotes

Environment

Jupyter Book: 0.8.3
MyST-NB: 0.10.1
Sphinx Book Theme: 0.0.38
MyST-Parser: 0.12.10
Jupyter-Cache: 0.4.1
NbClient: 0.5.0

[mmcky]

@chrisjsewell does markdown-it-py have access to file and lineno attributes? Can we add those to the Error as I have found migrating the quantecon lectures tricky to figure out where the syntax is failing when converting to notebooks.

[mmcky]

I'd be happy to work on this -- but need to get familiar with this repo so any pointers would be helpful if you don't have time.

[chrisjsewell]

@chrisjsewell does markdown-it-py have access to file and lineno attributes? Can we add those to the Error as I have found migrating the quantecon lectures tricky to figure out where the syntax is failing when converting to notebooks.

This is not a known error though, the parser should literally never raise an exception (thats how HTML/Markdown parsers work). Its only really possible to fail "gracefully" for known failures.

But I'll see if I can replicate the error now

[chrisjsewell]

FYI, you know this is the same exception as #50. Its probably just this function that needs to be fixed and you won't get anymore errors

[chrisjsewell]

I still can't replicate your exception.

I added to the tox.ini:

[testenv:py37-jupytext]
skip_install = true
deps = jupytext
commands = jupytext test.md --from myst --to ipynb

$ tox -r -e py37-jupytext
py37-jupytext recreate: /Users/chrisjsewell/Documents/GitHub/markdown-it-py/.tox/py37-jupytext
py37-jupytext installdeps: jupytext
py37-jupytext installed: appdirs==1.4.4,attrs==20.2.0,certifi==2020.6.20,flake8==3.8.3,importlib-metadata==2.0.0,ipython-genutils==0.2.0,jsonschema==3.2.0,jupyter-core==4.6.3,jupytext==1.6.0,markdown-it-py==0.5.5,mccabe==0.6.1,nbformat==5.0.8,pathspec==0.8.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyrsistent==0.17.3,PyYAML==5.3.1,regex==2020.7.14,six==1.15.0,toml==0.10.1,traitlets==5.0.5,typed-ast==1.4.1,zipp==3.3.1
py37-jupytext run-test-pre: PYTHONHASHSEED='3725018547'
py37-jupytext run-test: commands[0] | jupytext file.md --from myst --to ipynb
[jupytext] Reading test.md in format md:myst
[jupytext] Writing test.ipynb

Can you try this

[mmcky]

FYI, you know this is the same exception as #50. Its probably just this function that needs to be fixed and you won't get anymore errors

these blockquote issues are really confusing. The issue in #50 I was having was embedded literal in a block quote. The issue reported here is having two blockquotes in a document and trying to transfer them using jupytext. So maybe this issue isn't with markdown-it-py but the trace back is incorrect?

[mmcky]

@chrisjsewell I did the above suggestion -- and it is running. Grr -- so confused on this issue as it keeps cropping up in the migrations. I edit the file down to those block quotes -- maybe there is some hidden char causing the issue.

[chrisjsewell]

The phantom exception 😂
Yeh I have no doubt that it is an issue. But it would be really good to have a piece of text that raises the exception, otherwise it's obviously difficult to verify that any fix is working

[mmcky]

Ah -- I got it. If there are two blank lines at the end it fails.

  File "repos-collab/ebp/markdown-it-py/.tox/py37-jupytext/lib/python3.7/site-packages/markdown_it/rules_block/state_block.py", line 134, in skipEmptyLines
    if (self.bMarks[from_pos] + self.tShift[from_pos]) < self.eMarks[from_pos]:
IndexError: list index out of range
ERROR: InvocationError for command /repos-collab/ebp/markdown-it-py/.tox/py37-jupytext/bin/jupytext test.md --from myst --to ipynb (exited with code 1)
_______________________________________________________________________ summary _______________________________________________________________________
ERROR:   py37-jupytext: commands failed

[chrisjsewell]

There you go thats the one 👍

[mmcky]

but there is some interaction with having two (or more) blockquotes present as having one in the document works just fine with trailing new lines.

[chrisjsewell]

and you only need this to reproduce:

> A Block Quote

> Another Block Quote

[chrisjsewell]

done

[chrisjsewell]

Just wait to merge #59 then I'll make a release

[jackbentley]

Weirdly, in this very specific case, I still get this same error:

https://gist.github.com/jackbentley/3accfd5dcb4721d1466d461f9eeb3729

Note that the blockquotes need to be inside a list and the file has to have at least 3 blank lines at the end of the file. It does not matter if the blockquote is the last item in the file or not.