MyST lexer can't handle code fences with additional backticks

[davidselassie]

The following put in a MyST Markdown file causes a LexingException:

`````
this should lex
`````

With the following exception:

sybil.exceptions.LexingException: Could not match '(?<=\\n)```(:?\\n|\\Z)' in /home/dsel/test/test.md:
'`````\n'

Additional backticks are sometimes needed when you want to nest directives or handle literal code that has three backticks.

[cjw296]

Can you provide the sybil config you used to generate the above? A full exception would also be helpful...
(the reason I ask is the changes seem obvious, but I don't know what config you're using to hit a fenced code block without being a MyST directive, which is what the above appears to be...)

[davidselassie]

This is the relevant part of conftest.py.

import doctest

from sybil import Sybil
from sybil.parsers import myst

doctest_option_flags = doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE

pytest_collect_file = Sybil(
    parsers=[
        myst.PythonCodeBlockParser(doctest_optionflags=doctest_option_flags),
        myst.SkipParser(),
    ],
    patterns=["*.md", "*.py", "*.pyi"],
).pytest()

Here is one of the backtraces from pytest.

venv/3.10/lib/python3.10/site-packages/sybil/integration/pytest.py:118: in collect
    self.document = self.sybil.parse(self.path)
venv/3.10/lib/python3.10/site-packages/sybil/sybil.py:145: in parse
    return type_.parse(str(path), *self.parsers, encoding=self.encoding)
venv/3.10/lib/python3.10/site-packages/sybil/document.py:52: in parse
    for region in parser(document):
venv/3.10/lib/python3.10/site-packages/sybil/parsers/abstract/codeblock.py:73: in __call__
    for region in self.codeblock_parser(document):
venv/3.10/lib/python3.10/site-packages/sybil/parsers/abstract/codeblock.py:53: in __call__
    for region in self.lexers(document):
venv/3.10/lib/python3.10/site-packages/sybil/parsers/abstract/lexers.py:65: in __call__
    raise LexingException(
E   sybil.exceptions.LexingException: Could not match '(?<=\\n)```(:?\\n|\\Z)' in /home/dsel/bytewax/bytewax/docs/apidocs/bytewax/bytewax.connectors.demo.md:
E   ':canonical: bytewax.connectors.demo.RandomMetricSource.list_parts\n\n````\n\n````{py:method} build_part(now: datetime.datetime, for_part: str, resume_state: typing.Optional[bytewax.connectors.demo._RandomMetricState])\n:canonical: bytewax.connectors.demo.RandomMetricSource.build_part\n\n````\n\n`````\n\n'

Here is the contents of that file FWIW:

# {py:mod}`bytewax.connectors.demo`

```{py:module} bytewax.connectors.demo
```

```{autodoc2-docstring} bytewax.connectors.demo
:parser: myst
:allowtitles:
```

## Data

````{py:data} X
:canonical: bytewax.connectors.demo.X
:type: typing.TypeVar

```{autodoc2-docstring} bytewax.connectors.demo.X
:parser: myst
```

````


## Classes

`````{py:class} RandomMetricSource(metric_name: str, interval: datetime.timedelta = timedelta(seconds=0.7), count: int = sys.maxsize, next_random: typing.Callable[[], float] = lambda: random.randrange(0, 10))
:canonical: bytewax.connectors.demo.RandomMetricSource

:Bases:
    - {py:obj}`~bytewax.inputs.FixedPartitionedSource``[`{py:obj}`~typing.Tuple``[`{py:obj}`~str``, `{py:obj}`~float``], `{py:obj}`~bytewax.connectors.demo._RandomMetricState``]`

```{autodoc2-docstring} bytewax.connectors.demo.RandomMetricSource
:parser: myst
```

```{rubric} Initialization
```

```{autodoc2-docstring} bytewax.connectors.demo.RandomMetricSource.__init__
:parser: myst
```

````{py:method} list_parts() -> typing.List[str]
:canonical: bytewax.connectors.demo.RandomMetricSource.list_parts

````

````{py:method} build_part(now: datetime.datetime, for_part: str, resume_state: typing.Optional[bytewax.connectors.demo._RandomMetricState])
:canonical: bytewax.connectors.demo.RandomMetricSource.build_part

````

`````

[cjw296]

I'm glad I checked, my proposed fix in #113 doesn't work for this! Working on it now...

[cjw296]

@davidselassie - this should be fixed now, but I'll hold off releasing for now in the hope we can get #110 done too, but headsup that I'm likely to move onto another project and close that as "won't fix" if I don't hear back soon...

[cjw296]

Released here: https://pypi.org/project/sybil/6.1.0/