INTERNALERROR> SystemError: AST constructor recursion depth mismatch (before=78, after=133) #11724

coneybeare · 2023-08-10T19:01:42Z

Summary

We are seeing a very weird error in which coverage runs with html outputs are raising this INTERNALERROR

executed command and options:

pytest --cov app/ --cov-report term-missing --cov-report html:coverage --cov-branch --cov-append -vv

Error traceback

INTERNALERROR> Traceback (most recent call last):00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 270, in wrap_session00:14
INTERNALERROR>     session.exitstatus = doit(config, session) or 000:14
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 324, in _main00:14
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 433, in __call__00:14
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)00:14
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 112, in _hookexec00:14
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)00:14
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 133, in _multicall00:14
INTERNALERROR>     teardown[0].send(outcome)00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pytest_cov/plugin.py", line 307, in pytest_runtestloop00:14
INTERNALERROR>     self.cov_total = self.cov_controller.summary(self.cov_report)00:14
INTERNALERROR>                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper00:14
INTERNALERROR>     return meth(self, *args, **kwargs)00:14
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pytest_cov/engine.py", line 189, in summary00:14
INTERNALERROR>     total = self.cov.html_report(ignore_errors=True, directory=output)00:14
INTERNALERROR>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/control.py", line 1157, in html_report00:14
INTERNALERROR>     ret = reporter.report(morfs)00:14
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/html.py", line 313, in report00:14
INTERNALERROR>     self.write_html_file(ftr, prev_html, next_html)00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/html.py", line 388, in write_html_file00:14
 00:14
INTERNALERROR>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/html.py", line 130, in data_for_file00:14
INTERNALERROR>     for lineno, tokens in enumerate(fr.source_token_lines(), start=1):00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/phystokens.py", line 120, in source_token_lines00:14
INTERNALERROR>     match_case_lines = MatchCaseFinder(source).match_case_lines00:14
INTERNALERROR>                        ^^^^^^^^^^^^^^^^^^^^^^^00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/phystokens.py", line 85, in __init__00:14
INTERNALERROR>     self.visit(ast.parse(source))00:14
INTERNALERROR>                ^^^^^^^^^^^^^^^^^00:14
INTERNALERROR>   File "/usr/local/lib/python3.11/ast.py", line 50, in parse00:14
INTERNALERROR>     return compile(source, filename, mode, flags,00:14
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^00:14
INTERNALERROR> SystemError: AST constructor recursion depth mismatch (before=78, after=133)

We also cannot reproduce locally, only on CI (semaphore), very strange.

Expected vs actual result

I would expect the coverage not to raise this error. However I am not entirely convinced it is a bug, and not a config issue. Please educate me.

Reproducer

Versions

Output of relevant packages pip list, python --version, pytest --version etc.

pytest==7.4.0
pytest-cov==4.1.0

% python --version
Python 3.11.3

Make sure you include complete output of tox if you use it (it will show versions of various things).

Config

Include your tox.ini, pytest.ini, .coveragerc, setup.cfg or any relevant configuration.

Code

Link to your repository, gist, pastebin or just paste raw code that illustrates the issue.

If you paste raw code make sure you quote it, eg:

def foobar():
    pass

The text was updated successfully, but these errors were encountered:

coneybeare · 2023-08-10T20:16:00Z

We have isolated it comes from the --cov-report html:coverage as the error is not raised when that option is omitted.

coneybeare · 2023-08-10T22:21:43Z

And it seems to have to do with match_case_lines and code that uses it? We only have one place in this code for that

        match action:
            case foo.CREATED:
                do_something
            case foo.BLAH:
                pass
            case _:
                do_something_else

nedbat · 2023-08-11T01:45:45Z

This is weird. Any chance you can share the actual code? What version of coverage do you have installed? About the CI, can you check the precise versions of Python and the packages you are using?

coneybeare · 2023-08-14T16:39:06Z

even in CI, we run the tests in a container, so should be the same... perhaps semaphore is using something different 🤔 I can't explain it

Python 3.11.4

root@2306106a72eb:/rosetta# pip list
Package                           Version
--------------------------------- -----------
accumulation-tree                 0.6.2
alembic                           1.11.2
anyio                             3.7.0
appdirs                           1.4.4
astroid                           2.15.6
asyncpg                           0.28.0
attrs                             23.1.0
black                             23.7.0
boto3                             1.28.24
botocore                          1.31.24
callee                            0.3.1
cattrs                            23.1.2
certifi                           2023.7.22
cffi                              1.15.1
chardet                           5.2.0
charset-normalizer                3.2.0
click                             8.1.6
colorama                          0.4.6
colorlog                          6.6.0
coverage                          7.2.7
cryptography                      41.0.3
dill                              0.3.7
dnspython                         2.4.2
ecs-logging                       1.1.0
email-validator                   2.0.0.post2
factory-boy                       3.3.0
Faker                             19.3.0
fastapi                           0.101.0
fastapi-route-logger-middleware   0.2.3
flake8                            6.1.0
greenlet                          2.0.2
gunicorn                          20.1.0
h11                               0.14.0
httpcore                          0.17.3
httptools                         0.5.0
httpx                             0.24.1
idna                              3.4
inflection                        0.5.1
iniconfig                         2.0.0
isort                             5.12.0
Jinja2                            3.1.2
jmespath                          1.0.1
lazy-object-proxy                 1.9.0
Mako                              1.2.4
markdown-it-py                    3.0.0
MarkupSafe                        2.1.3
marshmallow                       3.20.1
marshmallow-jsonapi               0.24.0
mccabe                            0.7.0
mdurl                             0.1.2
multidict                         6.0.4
mypy                              1.5.0
mypy-extensions                   1.0.0
numpy                             1.25.2
ochrona                           2.0.2
packaging                         23.1
pandas                            2.0.3
parameterized                     0.9.0
pathspec                          0.11.2
pip                               23.1.2
platformdirs                      3.10.0
pluggy                            1.2.0
pretty-errors                     1.2.25
prometheus-client                 0.17.1
prometheus-fastapi-instrumentator 6.1.0
psycopg2-binary                   2.9.7
py                                1.11.0
pybrake                           1.10.1
pycodestyle                       2.11.0
pycparser                         2.21
pydantic                          1.10.10
pydantic-jsonapi                  0.11.0
pydocstyle                        6.3.0
pyflakes                          3.1.0
Pygments                          2.16.1
pyinstrument                      4.5.1
pylint                            2.17.5
pyparsing                         3.1.1
pytest                            7.4.0
pytest-cov                        4.1.0
python-dateutil                   2.8.2
python-dotenv                     1.0.0
python-multipart                  0.0.6
pytz                              2023.3
pyudorandom                       1.0.0
PyYAML                            6.0
redis                             4.6.0
requests                          2.31.0
requests-cache                    1.1.0
rich                              13.5.2
s3transfer                        0.6.1
semver                            3.0.1
setuptools                        65.5.1
simplejson                        3.19.1
six                               1.16.0
sniffio                           1.3.0
snowballstemmer                   2.2.0
SQLAlchemy                        2.0.19
SQLAlchemy-Utils                  0.41.1
starlette                         0.27.0
starlette-context                 0.3.6
tarsafe                           0.0.5
tdigest                           0.5.2.2
toml                              0.10.2
tomlkit                           0.12.1
types-pyOpenSSL                   23.2.0.2
types-redis                       4.6.0.3
types-requests                    2.31.0.2
types-simplejson                  3.19.0.2
types-urllib3                     1.26.25.14
typing_extensions                 4.6.3
tzdata                            2023.3
url-normalize                     1.4.3
urllib3                           1.26.16
uvicorn                           0.23.2
uvloop                            0.17.0
vcrpy                             5.1.0
vcrpy-unittest                    0.1.7
watchfiles                        0.19.0
websockets                        11.0.3
wheel                             0.40.0
wrapt                             1.15.0
yarl                              1.9.2

root@2306106a72eb:/rosetta# pytest --cov app/ --cov-report term-missing --cov-report html:coverage --cov-branch --cov-append -vv
================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /rosetta
plugins: Faker-19.3.0, cov-4.1.0, anyio-3.7.0
collected 402 items

... <ALL PASSING TESTS> ...

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 433, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 112, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 133, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pytest_cov/plugin.py", line 307, in pytest_runtestloop
INTERNALERROR>     self.cov_total = self.cov_controller.summary(self.cov_report)
INTERNALERROR>                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/pytest_cov/engine.py", line 189, in summary
INTERNALERROR>     total = self.cov.html_report(ignore_errors=True, directory=output)
INTERNALERROR>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/control.py", line 1157, in html_report
INTERNALERROR>     ret = reporter.report(morfs)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/html.py", line 313, in report
INTERNALERROR>     self.write_html_file(ftr, prev_html, next_html)
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/html.py", line 388, in write_html_file
INTERNALERROR>     file_data = self.datagen.data_for_file(ftr.fr, ftr.analysis)
INTERNALERROR>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/html.py", line 130, in data_for_file
INTERNALERROR>     for lineno, tokens in enumerate(fr.source_token_lines(), start=1):
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/phystokens.py", line 120, in source_token_lines
INTERNALERROR>     match_case_lines = MatchCaseFinder(source).match_case_lines
INTERNALERROR>                        ^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/site-packages/coverage/phystokens.py", line 85, in __init__
INTERNALERROR>     self.visit(ast.parse(source))
INTERNALERROR>                ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/local/lib/python3.11/ast.py", line 50, in parse
INTERNALERROR>     return compile(source, filename, mode, flags,
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> SystemError: AST constructor recursion depth mismatch (before=78, after=133)

=========================================================================================================================== 402 passed, 62 warnings in 21.74s ============================================================================================================================

# echo $?
3

coneybeare · 2023-08-14T16:46:33Z

We have 2 places we use match/case

class FooClassHelper:
    @staticmethod
    async def foo_method(
        action: "MappingAction",
        msg: str = '',
        request_id: str = '',
        mapping_attributes: Optional[MappingAttributes] = None,
        mapping_attributes_array: Optional[list[Union[MappingAttributes, None]]] = None,
        username: str = 'steve',
        **kwargs: Any,
    ) -> None:
       event_mapping_data =  {} # await, response from another method
        match action:
            case MappingAction.CREATED:
                if not mapping_attributes_array:
                    Logger.warning(...)
            case MappingAction.MERGED:
                pass
            case _:
                if not event_mapping_data:
                    Logger.warning(...)

and


@router.post("/{api_version}/foos")
async def create_foo_mapping(
        api_version: ApiVersion,
        request: ApplicationRequest,
        foo_attributes: FooAttributes,
) -> Union[Dict, None]:
    match foo_attributes.foo_type:
        case EntityTypes.FOO:
            return await handle_request(...)

dpritchett-rebellion · 2023-09-07T17:57:53Z

Note to future error googlers: We're seeing this same exception despite not even using pytest-cov. I managed to work around it by telling pytest to use a less-aggressive traceback format: pytest --tb=line gets me shorter output on failures but at least it doesn't hang up in the AST recursion wilderness.

We do have a few match/case statements in our codebase, and we've recently upgraded from Python 3.10 to 3.11.

Notes on --tb options: https://docs.pytest.org/en/7.1.x/how-to/output.html

coneybeare · 2023-09-11T14:52:47Z

Nice find, this worked for me as a workaround too

wbloos · 2023-09-22T11:53:53Z

@dpritchett-rebellion should this issue be moved to the main project to get the attention it needs?

Andrew-Brock · 2023-12-18T02:02:40Z

I was facing this same error too with PyTest on Python 3.11.7. The exact same PyTest version and test suite on Python versions 3.8, 3.9 & 3.10 did not have this issue. I am using pytest-cov, but disabling it did not seem to help.

For me, adding --tb=native (see documentation) made the error go away and also still means I get callstacks for any failures as opposed to the previously suggested --tb=line.

I spent some time trying to figure out what the actual problem was and trying different combinations, but didn't want to spend too much time.
A few observations I did make though:

--tb=line did not seem to help my situation. It meant that PyTest didn't crash and abort in the middle of my test suite, but it was still encountering a bogus failure with SystemError: AST constructor recursion depth mismatch (before=174, after=210) in the middle of 1 of my tests and hence making it fail.
--tb=native is not creating bogus failures in my tests
Despite running the exact same container with all of the same Python, dependencies, etc. in Docker for Windows (in WLS2 mode) and on Ubuntu, it was only failing in Ubuntu. My git is configured to always check out with LF line endings so in theory this should be identical between WIndows & Linux but there must be something different. I was also not able to reproduce it locally on Windows outside of Docker either, despite also having the exact same Python 3.11.7 & requirements installed.

brianzi · 2024-01-16T17:02:33Z

I encountered this problem today and I'm pretty sure the cause is a bug in the parser of cpython itself (python/cpython#106905), which has been around in that form in python3.11 releases since 3.11.0b4 up to and including 3.11.7.

But this bug has now been fixed though, and indeed, for me, switching from 3.11.7 to a post-3.11.7 build has removed the issue. That bugfix is also listed already in the announcement for the next 3.11 release

Hope this helps!

Zac-HD · 2024-02-17T20:03:03Z

Thank you @brianzi! This does indeed seem to be an upstream CPython bug, which has been fixed and released in 3.11.8.

phofl mentioned this issue Dec 12, 2023

CI fails sometime - pytest problem dask/dask-expr#520

Closed

Pierre-Sassoulas transferred this issue from pytest-dev/pytest-cov Dec 18, 2023

Woody1193 mentioned this issue Jan 5, 2024

AST Recursion Depth Mismatch on error in setup_for_execution dagster-io/dagster#19047

Open

Zac-HD closed this as completed Feb 17, 2024

Zac-HD added type: bug problem that needs to be addressed topic: rewrite related to the assertion rewrite mechanism labels Feb 17, 2024

eellison mentioned this issue May 2, 2024

Weird AST constructor issue with mode="max-autotune" with python 3.11 pytorch/pytorch#125374

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

INTERNALERROR> SystemError: AST constructor recursion depth mismatch (before=78, after=133) #11724

INTERNALERROR> SystemError: AST constructor recursion depth mismatch (before=78, after=133) #11724

coneybeare commented Aug 10, 2023

coneybeare commented Aug 10, 2023

coneybeare commented Aug 10, 2023

nedbat commented Aug 11, 2023

coneybeare commented Aug 14, 2023

coneybeare commented Aug 14, 2023 •

edited

dpritchett-rebellion commented Sep 7, 2023 •

edited

coneybeare commented Sep 11, 2023

wbloos commented Sep 22, 2023

Andrew-Brock commented Dec 18, 2023

brianzi commented Jan 16, 2024

Zac-HD commented Feb 17, 2024

INTERNALERROR> SystemError: AST constructor recursion depth mismatch (before=78, after=133) #11724

INTERNALERROR> SystemError: AST constructor recursion depth mismatch (before=78, after=133) #11724

Comments

coneybeare commented Aug 10, 2023

Summary

Expected vs actual result

Reproducer

Versions

Config

Code

coneybeare commented Aug 10, 2023

coneybeare commented Aug 10, 2023

nedbat commented Aug 11, 2023

coneybeare commented Aug 14, 2023

coneybeare commented Aug 14, 2023 • edited

dpritchett-rebellion commented Sep 7, 2023 • edited

coneybeare commented Sep 11, 2023

wbloos commented Sep 22, 2023

Andrew-Brock commented Dec 18, 2023

brianzi commented Jan 16, 2024

Zac-HD commented Feb 17, 2024

coneybeare commented Aug 14, 2023 •

edited

dpritchett-rebellion commented Sep 7, 2023 •

edited