AssertionError: Internal error: method must be called on parsed file only

[daisuke834]

Bug Report

When I run mypy==0.991 in parallel, the following error randomly happens.

Traceback (most recent call last):
  File "/home/my_name/.pyenv/versions/3.9.16/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/my_name/.pyenv/versions/3.9.16/lib/python3.9/site-packages/mypy/__main__.py", line 15, in console_entry
    main()
  File "mypy/main.py", line 95, in main
  File "mypy/main.py", line 174, in run_build
  File "mypy/build.py", line 193, in build
  File "mypy/build.py", line 276, in _build
  File "mypy/build.py", line 2903, in dispatch
  File "mypy/build.py", line 3284, in process_graph
  File "mypy/build.py", line 3365, in process_fresh_modules
  File "mypy/build.py", line 2109, in fix_cross_refs
AssertionError: Internal error: method must be called on parsed file only

To Reproduce

I cannot identify to reproduce this problem because this error randomly happens (2-3% probabiliy), but I observed this problem only when I ran mypy in parallel on multiple files.

Your Environment

• Mypy version used: mypy==0.991
• Mypy command-line flags: mypy --follow-imports silent --config-file pyproject.toml path/to/foo.py
• Mypy configuration options from mypy.ini (and other config files): pyproject.toml

[tool.mypy]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = "third_party.*"
ignore_errors = true
[[tool.mypy.overrides]]
module = "requests.*"
ignore_missing_imports = true

• Python version used:3.9.16

[Alex031544]

Same here...
It seems, that it occurs on files without findings.

E.g.: I put a faulty code in a file, where I assume the source looks fine:

a = dict()
for d in a:
    print(d)

... and the AssertionError disappears!
But this might be only the half truth as some files pass anyhow. Maybe this is something in conjunction with the cache.

---

I found out, that this issue occurs in dependency with the cache. If several files are checked this leads to trouble. Using separate cache dirs helped.

[hauntsaninja]

@Alex031544 it seems your issue might be different from OP's, since it sounds like you're only running one mypy command at a time. If you can provide enough details that we could reproduce your issue, that would be much appreciated!

[Alex031544]

@Alex031544 it seems your issue might be different from OP's, since it sounds like you're only running one mypy command at a time. If you can provide enough details that we could reproduce your issue, that would be much appreciated!

That's how I call mypy as part of one unit test with one subcase per file - Therefore I am running several mypy tests parrallel:

import os
import glob
import multiprocessing
import re
import shutil
import subprocess
import unittest
from dataclasses import dataclass
from typing import List


@dataclass
class Result:
    return_val: int
    stdout: str
    stderr: str


class StaticTest(unittest.TestCase):

    mypy_cache_based_dir = '.mypy_cache'

    def _worker_mypy(self, filename: str, results: dict):
        print('mypy on ' + filename)
        cache_dir = self.mypy_cache_based_dir + '/' + filename
        cache_dir = re.sub(r'[^\w./]', '_', cache_dir)

        call: List[str] = ['.venv/bin/mypy'] \
                          + self.mypy_opts \
                          + ['--cache-dir %s' % cache_dir] \
                          + [filename]
        p = subprocess.Popen(' '.join(call),
                             shell=True,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
        stdout, stderr = p.communicate()
        p.wait()
        results[filename] = Result(p.returncode, stdout.decode(), stderr.decode())

    def test_run_mypy_tests(self):
        """
        Run MyPy on all files in search paths
        """

        os.mkdir(self.mypy_cache_based_dir)

        jobs = []
        manager = multiprocessing.Manager()
        results = manager.dict()
        for test_file in self.test_files:
            p = multiprocessing.Process(target=self._worker_mypy, args=(test_file, results,))
            jobs.append(p)
            p.start()

        for p in jobs:
            p.join()

        for key, result in results.items():
            with self.subTest(msg=f'mypy check on {key}'):
                self.assertEqual(0, result.return_val,
                                 f'mypy failed on {key}\n\n{result.stdout}\n\n{result.stderr}')
        shutil.rmtree(self.mypy_cache_based_dir, ignore_errors=True)

    def __init__(self, *args, **kwargs) -> None:
        self.pkgname: str = "browse"
        super(StaticTest, self).__init__(*args, **kwargs)
        my_env = os.environ.copy()
        self.pypath: str = my_env.get("PYTHONPATH", os.getcwd())

        self.mypy_opts: List[str] = ['--config-file .mypyrc']

        search_paths = ['pylib', 'pytest']
        self.test_files: List[str] = []
        for spath in search_paths:
            self.test_files.extend(glob.iglob(f'{os.getcwd()}/{spath}/**/*.py', recursive=True))


if __name__ == '__main__':
    unittest.main()

Now for each call respectively file under test, a unique path is created below '.mypy_cache'. The cache folder will be deleted when finished at once.

[daisuke834]

This issue was reproduced with 1.0.0 as well.

[gresavage]

I was consistently having this issue as well with mypy 0.993 when running as a pre-commit hook. As with @Alex031544 it was only occurring on files without findings.

I upgraded to 1.4.1 and it went away. Not sure of which versions between 1.0.0 (per @daisuke834) and 1.4.1 also fix the issue.

[gresavage]

I was consistently having this issue as well with mypy 0.993 when running as a pre-commit hook. As with @Alex031544 it was only occurring on files without findings.

I upgraded to 1.4.1 and it went away. Not sure of which versions between 1.0.0 (per @daisuke834) and 1.4.1 also fix the issue.

nevermind - I am still having the issue in 1.4.1 😮‍💨

[gresavage]

Sorry to bombard the thread - I am using a conda (specifically mamba) environment and found the following mypy info:

mypy                      1.4.1           py310h2372a71_0    conda-forge
mypy_extensions           1.0.0              pyha770c72_0    conda-forge

I then tried force installing mypy==1.4.1 from pip and retried. Lo and behold, the errors went away. So I wonder if it might be localized to the distributions on conda-forge???

[JelleZijlstra]

Possibly related to whether or not mypy is compiled with mypyc?

[lucemia]

As for me, it seems like the problem is connected to --install-types.

[LukasJerabek]

We experience similar issues with the same traceback, in our case it happens only if we add MYPYPATH=./src (which is our root source codes directory) in front of mypy call. When we remove it, problems disappear.

[peterhessey]

I was also having this issue when using mypy in a pre-commit hook with repo: local. Adding require_serial: true to the hook fixed this for me!

[LukasJerabek]

Our case was fixed by removing sqlalchemy from plugins from .mypy.ini
from this:
plugins = pydantic.mypy, sqlalchemy.ext.mypy.plugin
to this:
plugins = pydantic.mypy