Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MyST lexer can't handle code fences with additional backticks #111

Closed
davidselassie opened this issue Feb 2, 2024 · 5 comments · Fixed by #113
Closed

MyST lexer can't handle code fences with additional backticks #111

davidselassie opened this issue Feb 2, 2024 · 5 comments · Fixed by #113

Comments

@davidselassie
Copy link

The following put in a MyST Markdown file causes a LexingException:

`````
this should lex
`````

With the following exception:

sybil.exceptions.LexingException: Could not match '(?<=\\n)```(:?\\n|\\Z)' in /home/dsel/test/test.md:
'`````\n'

Additional backticks are sometimes needed when you want to nest directives or handle literal code that has three backticks.

@cjw296
Copy link
Member

cjw296 commented Feb 9, 2024

Can you provide the sybil config you used to generate the above? A full exception would also be helpful...
(the reason I ask is the changes seem obvious, but I don't know what config you're using to hit a fenced code block without being a MyST directive, which is what the above appears to be...)

@cjw296 cjw296 linked a pull request Feb 9, 2024 that will close this issue
@davidselassie
Copy link
Author

This is the relevant part of conftest.py.

import doctest

from sybil import Sybil
from sybil.parsers import myst

doctest_option_flags = doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE

pytest_collect_file = Sybil(
    parsers=[
        myst.PythonCodeBlockParser(doctest_optionflags=doctest_option_flags),
        myst.SkipParser(),
    ],
    patterns=["*.md", "*.py", "*.pyi"],
).pytest()

Here is one of the backtraces from pytest.

venv/3.10/lib/python3.10/site-packages/sybil/integration/pytest.py:118: in collect
    self.document = self.sybil.parse(self.path)
venv/3.10/lib/python3.10/site-packages/sybil/sybil.py:145: in parse
    return type_.parse(str(path), *self.parsers, encoding=self.encoding)
venv/3.10/lib/python3.10/site-packages/sybil/document.py:52: in parse
    for region in parser(document):
venv/3.10/lib/python3.10/site-packages/sybil/parsers/abstract/codeblock.py:73: in __call__
    for region in self.codeblock_parser(document):
venv/3.10/lib/python3.10/site-packages/sybil/parsers/abstract/codeblock.py:53: in __call__
    for region in self.lexers(document):
venv/3.10/lib/python3.10/site-packages/sybil/parsers/abstract/lexers.py:65: in __call__
    raise LexingException(
E   sybil.exceptions.LexingException: Could not match '(?<=\\n)```(:?\\n|\\Z)' in /home/dsel/bytewax/bytewax/docs/apidocs/bytewax/bytewax.connectors.demo.md:
E   ':canonical: bytewax.connectors.demo.RandomMetricSource.list_parts\n\n````\n\n````{py:method} build_part(now: datetime.datetime, for_part: str, resume_state: typing.Optional[bytewax.connectors.demo._RandomMetricState])\n:canonical: bytewax.connectors.demo.RandomMetricSource.build_part\n\n````\n\n`````\n\n'

Here is the contents of that file FWIW:

# {py:mod}`bytewax.connectors.demo`

```{py:module} bytewax.connectors.demo
```

```{autodoc2-docstring} bytewax.connectors.demo
:parser: myst
:allowtitles:
```

## Data

````{py:data} X
:canonical: bytewax.connectors.demo.X
:type: typing.TypeVar

```{autodoc2-docstring} bytewax.connectors.demo.X
:parser: myst
```

````


## Classes

`````{py:class} RandomMetricSource(metric_name: str, interval: datetime.timedelta = timedelta(seconds=0.7), count: int = sys.maxsize, next_random: typing.Callable[[], float] = lambda: random.randrange(0, 10))
:canonical: bytewax.connectors.demo.RandomMetricSource

:Bases:
    - {py:obj}`~bytewax.inputs.FixedPartitionedSource``[`{py:obj}`~typing.Tuple``[`{py:obj}`~str``, `{py:obj}`~float``], `{py:obj}`~bytewax.connectors.demo._RandomMetricState``]`

```{autodoc2-docstring} bytewax.connectors.demo.RandomMetricSource
:parser: myst
```

```{rubric} Initialization
```

```{autodoc2-docstring} bytewax.connectors.demo.RandomMetricSource.__init__
:parser: myst
```

````{py:method} list_parts() -> typing.List[str]
:canonical: bytewax.connectors.demo.RandomMetricSource.list_parts

````

````{py:method} build_part(now: datetime.datetime, for_part: str, resume_state: typing.Optional[bytewax.connectors.demo._RandomMetricState])
:canonical: bytewax.connectors.demo.RandomMetricSource.build_part

````

`````

@cjw296
Copy link
Member

cjw296 commented Mar 12, 2024

I'm glad I checked, my proposed fix in #113 doesn't work for this! Working on it now...

@cjw296
Copy link
Member

cjw296 commented Mar 18, 2024

@davidselassie - this should be fixed now, but I'll hold off releasing for now in the hope we can get #110 done too, but headsup that I'm likely to move onto another project and close that as "won't fix" if I don't hear back soon...

@cjw296
Copy link
Member

cjw296 commented Apr 22, 2024

Released here: https://pypi.org/project/sybil/6.1.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants