Windows Python 3.5/3.6 x64 issues

[esoma]

Specifically, within the environment that the github workflows uses for windows with Python 3.5/3.6 x64, there are strange issues when trying to mypycify a module -- either a DLL load failure or memory access violations. I first encountered this in my own project's workflows when adding some tests for mypyc integration, when attempting to mypycify a file I was getting memory access violations (https://github.com/esoma/pgo/runs/2155764031).

I tried to replicate the environment as best I could on my Windows 10 machine, but have been unable to reproduce. I created a reproducer repo (https://github.com/esoma/mypy-crash) which doesn't produce memory access violations, but instead has a DLL load failure (https://github.com/esoma/mypy-crash/runs/2155914325).

I tried importing from mypyc.primitives.set_ops import new_set_op (the offending line in the previous DLL load failure) directly to see if that would immediately cause issues, but it (kind of expectedly) "moves" the DLL load failure elsewhere (https://github.com/esoma/mypy-crash/runs/2155962793).

As expected, if using a pure python version of mypy then we don't have any issues (https://github.com/esoma/mypy-crash/runs/2155996841). This (educated guess) is also why it works fine on x86 -- no windows x86 compiled wheels exist on pypi so those are already using pure python. I assume that if there were such wheels they would also fail in this circumstance.

Compiling mypy with mypyc on the machine itself rather than using the precompiled version from pypi will not result in a crash (https://github.com/esoma/mypy-crash/runs/2156078099).

For further info I tried installing older versions of mypy (https://github.com/esoma/mypy-crash/runs/2156164244). The issue starts in 0.761 and is mysteriously fixed in 0.800 (but then returns to failing in the latest release).

Next I tested that 3.5 also failed (once again only for x64) and then the earliest versions of 3.5 and 3.6 through the setup-python github action (3.5.4 and 3.6.7).

Here is the latest run that should demonstrate all of the above: https://github.com/esoma/mypy-crash/runs/2156224471

Without an environment to debug in, I can only guess at the actual problem though.

[JukkaL]

Can you try the latest development wheels from here: https://github.com/mypyc/mypy_mypyc-wheels/releases

We made some changes to the wheel builds recently, and I'm not sure if 0.812 include them. If they work, we can try releasing 0.813 built using the latest build scripts.

As a workaround, you should be able to force the installation of the pure Python package on x64. I don't remember exactly how, but maybe using --no-binary?

[esoma]

Thanks for the quick response.

Can you try the latest development wheels

Got the same result with the bfc67b6129b3f82a7a6e92eddeedbcfb70440a31 dev wheels:
3.5: https://github.com/esoma/mypy-crash/runs/2159728629
3.6: https://github.com/esoma/mypy-crash/runs/2159728301

As a workaround, you should be able to force the installation of the pure Python package on x64

Yup, that's my current solution.

[JukkaL]

Thanks for trying the dev wheels!

Here's a copy of the traceback:

Traceback (most recent call last):
  File "crash.py", line 2, in <module>
    from mypyc.build import mypycify
  File "mypyc\build.py", line 43, in <module>
  File "mypyc\codegen\emitmodule.py", line 22, in <module>
  File "mypyc\irbuild\main.py", line 40, in <module>
  File "mypyc\irbuild\builder.py", line 61, in <module>
  File "mypyc\irbuild\ll_builder.py", line 60, in <module>
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

There's been a similar issue with tensorflow: tensorflow/tensorflow#36167 (comment)

Based on the above issue, plausibly these could be causing the issue:

• The right MSVC redistributable is not available.
• Mix of 32-bit Python and 64-bit compiled module, or vice versa, somehow. This will clearly not work.

Can you check if one of above could be causing the issue?

[ichard26]

FWIW, I'm hitting this as well trying to compile Black with mypyc with cibuildbuild on Windows: https://github.com/ichard26/black-mypyc-wheels/runs/2964228506?check_suite_focus=true

And just in case the build logs have expired: windows-3.6-logs.txt. Here's a list of the provided software too.

For the time being, I'll try the interpreted workaround and see if it still works.

[JukkaL]

It would be useful to have some more information to narrow this down:

• Does this only impact Python 3.5/3.6 (do things work on 3.7+)? (Note that the next release of mypyc won't support Python 3.5.)
• A small example that reproduces the issue would simplify debugging.

[esoma]

I haven't touched this in awhile, but 3.7+ was working fine. I was running mypyc on an empty file and having the issue, so I don't think the contents is relevant.

Edit, forgot I'd made a reproducer repo: https://github.com/esoma/mypy-crash
It's a little messy from things I was testing out.