Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native parser crashes on pydoc_data.topics #980

Open
HMate opened this issue Jul 25, 2023 · 0 comments
Open

Native parser crashes on pydoc_data.topics #980

HMate opened this issue Jul 25, 2023 · 0 comments
Labels
parsing Converting source code into CST nodes

Comments

@HMate
Copy link

HMate commented Jul 25, 2023

I am trying to parse in the source files of the standard library. When I get to Lib/pydoc_data/topics.py I encounter issues.
The native parser of LibCST causes stack overflow on this file. It can be parsed correctly with the pure parser.
I am not sure how to debug it, but this file contains thousands of lines of ConcatenatedString nodes, so I would guess that causes the problem. If I use the pure parser, the code will error anyway on the cst.code expression with a RecursionError.

Tested on windows 10 with LibCST 1.0.1 on Python 3.11.4.

Code to reproduce:

import sys, os, faulthandler
from pathlib import Path
import libcst

# os.environ["LIBCST_PARSER_TYPE"] = "pure"
content = (Path(sys.base_prefix) / "Lib/pydoc_data/topics.py").read_text("utf-8")
faulthandler.enable()
cst: libcst.Module = libcst.parse_module(content)
print(cst.code)

Result of the run:

Windows fatal exception: stack overflow

Current thread 0x000020f8 (most recent call first):
  File "D:\projects\testing\libcst_error\.venv\Lib\site-packages\libcst\_parser\entrypoints.py", line 55 in _parse
  File "D:\projects\testing\libcst_error\.venv\Lib\site-packages\libcst\_parser\entrypoints.py", line 109 in parse_module
  File "D:\projects\testing\libcst_error\main.py", line 8 in <module>
@zsol zsol added the parsing Converting source code into CST nodes label Sep 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
parsing Converting source code into CST nodes
Projects
None yet
Development

No branches or pull requests

2 participants