New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Universal CRT not found building with Py3.5 on Win10 (Python 3.5 not usable on Windows) #1566

Closed
AcidWeb opened this Issue Oct 1, 2015 · 84 comments

Comments

Projects
None yet
@AcidWeb

AcidWeb commented Oct 1, 2015

PyInstaller: 3.0.dev8
Python: 3.5.0 64bit
Platform: Windows-10.0.10240

Freezing any code in this environment output some warnings:
https://gist.githubusercontent.com/AcidWeb/aa423c3f402cb86f1806/raw/d0bc853ffe262a4dfccef9febf0fac0b086aa458/gistfile1.txt

This is output of pyinstaller test.py where test.py is empty file(!).

@htgoebel htgoebel added the Windows label Oct 1, 2015

@matysek

This comment has been minimized.

Member

matysek commented Oct 1, 2015

@AcidWeb Thanks for reporting. Those warnings should be harmless.

I think PyInstaller should just ignore them.

@codewarrior0 Is it safe to ignore dlls with names like:

WARNING: lib not found: api-ms-win-crt-heap-l1-1-0.dll dependency of c:\python35\python.exe
2044 WARNING: lib not found: api-ms-win-crt-stdio-l1-1-0.dll dependency of c:\python35\python.exe
2068 WARNING: lib not found: api-ms-win-crt-math-l1-1-0.dll dependency of c:\python35\python.exe
2087 WARNING: lib not found: api-ms-win-crt-runtime-l1-1-0.dll dependency of c:\python35\python.exe
@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 1, 2015

Probably not. Those DLLs are part of the Universal CRT that Python 3.5 uses. I had to add them to the DLL whitelist to make sure they get included. They should be installed by the Python 3.5 installer They are installed as part of Windows 10 itself so I'm curious why they aren't found... maybe it's related to Windows 10, or to having them installed via Windows Update. instead of by the Python 3.5 installer.

@matysek

This comment has been minimized.

Member

matysek commented Oct 1, 2015

@codewarrior0 Are you going to fix this for 3.0?

@matysek

This comment has been minimized.

Member

matysek commented Oct 1, 2015

Or does it require more investigation?

@AcidWeb

This comment has been minimized.

AcidWeb commented Oct 1, 2015

Well I don't have any api-ms-win-* DLL files. And I got entire Visual Studio installed here.

Both Python and files created by PyInstaller work correctly.

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 1, 2015

Well I don't have any api-ms-win-* DLL files.
Both Python and files created by PyInstaller work correctly.

At least one of these must be false. Would you mind using Dependency Walker on your python.exe and tell me where it is loading the api-ms-win-crt-*.DLL files from?

Both Python and files created by PyInstaller work correctly.

This is expected. Even if PyInstaller doesn't bundle the CRT files, the built app will still run on your system, since you have the CRT files installed system wide. I'd also expect the built app to fail when moved to a system that doesn't have the Universal CRT installed.

@matysek

This comment has been minimized.

Member

matysek commented Oct 1, 2015

@codewarrior0 Are those warnings harmless in general?

@AcidWeb

This comment has been minimized.

AcidWeb commented Oct 1, 2015

Dependency Walker say they are missing.

Tomorrow I will check it on some clean Windows 10 environments.

@matysek

This comment has been minimized.

Member

matysek commented Oct 1, 2015

@codewarrior0 Moving this issue to 3.1 milestone. If you think it should be in 3.0 do otherwise.

@matysek matysek added the @medium label Oct 1, 2015

@matysek matysek added this to the PyInstaller 3.1 milestone Oct 1, 2015

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 1, 2015

@codewarrior0 Are you going to fix this for 3.0?

Needs more investigation. I don't even know where to begin fixing it.

@codewarrior0 Are those warnings harmless in general?

I wouldn't say they're harmless, but I don't know if making them fatal is right, either. The built app still works as long as the CRT is installed at the system level, after all.

Dependency Walker say they are missing.

That's just baffling.

I can't reproduce the warnings on my Windows 7 machine. I wonder if Windows 10 is doing some kind of magic to load those DLLs from a location that's invisible to both Dependency Walker and PyInstaller.

@AcidWeb Since I don't have Windows 10, do you feel like searching your entire Windows folder for the file api-ms-win-crt-runtime-l1-1-0.dll for me?

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 1, 2015

Tomorrow I will check it on some clean Windows 10 environments.

Any more info you can get would be helpful. Thanks.

@AcidWeb

This comment has been minimized.

AcidWeb commented Oct 1, 2015

Eh apparently wildcard search don't work on Windows 10. That is reason why I not found any -.-

Still I don't have any in Windows directory. But i have multiple copies as part of VS and Windows SDK.

I'm going to sleep now - I will try provide some additional information tomorrow.

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 1, 2015

For the record, I was wrong about the Python installer earlier. It doesn't install the CRT itself; instead, I had to install The UCRT Release for Windows Vista,7,8 from Microsoft before installing Python 3.5. This release isn't needed on Windows 10 because the UCRT is preinstalled.

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 1, 2015

@matysek Since this isn't fixed for 3.0, is there somewhere we can note this as a known issue?

"Apps built with Windows 10 and Python 3.5 may not run on Windows versions earlier than 10."

@matysek

This comment has been minimized.

Member

matysek commented Oct 1, 2015

@codewarrior0 I think either README.rst or doc/CHANGES.rst.

@AcidWeb

This comment has been minimized.

AcidWeb commented Oct 2, 2015

@codewarrior0
Clean Windows 10 - My executable work. Dependency Walker say that api-ms-win-* are missing. Not found that DDLs anywhere.

Sadly I don't have access to any clean 64bit Windows 7/8 enviroments. But I tested it on random computers without Python 3.5 and as expected they crash due missing DLLs files. But Dependency Walker display only few missing DLL files. Much more are missing on Windows 10.
https://i.iosphe.re/i/AcidWeb/jwcge2fzji1x

This is the ultimate proof that Win10 was created by black magic! :-P

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 2, 2015

This is the ultimate proof that Win10 was created by black magic! :-P

I am intrigued and terrified. Someday I'll have to install Win10 myself to get to the bottom of this. All I know is, Microsoft says those DLLs are included with Win10 by default, and I know I have them on Win7 because I installed the UCRT package.

@AcidWeb

This comment has been minimized.

AcidWeb commented Oct 2, 2015

You can use ModernIE VM for Win10 tests: https://dev.modern.ie/tools/vms/

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 2, 2015

Thanks! I almost forgot about ModernIE.

@AcidWeb

This comment has been minimized.

AcidWeb commented Oct 2, 2015

So I guess for now a "fix" is bundling proper redist packages with application installer.

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 2, 2015

Yep. Here's the link for the redist needed for Python 3.5:

https://www.microsoft.com/en-us/download/details.aspx?id=48234

@AcidWeb

This comment has been minimized.

AcidWeb commented Oct 2, 2015

Separate installer for every windows release? That is major pain in the ass.

The Visual Studio 2015 VCRedist package includes the above mentioned Windows Update packages, so simply installing the VCRedist will install both the Visual C++ libraries and the Universal CRT. This is our recommended deployment mechanism. On Windows XP, for which there is no Universal CRT Windows Update MSU, the VCRedist will deploy the Universal CRT itself.

So this is proper multi-os package:
https://www.microsoft.com/en-US/download/details.aspx?id=48145

@AcidWeb

This comment has been minimized.

AcidWeb commented Oct 2, 2015

@codewarrior0 @matysek Just confirming Visual C++ Redistributable for Visual Studio 2015 contain all missing DLLs and for now must be distributed with PyInstaller output when Windows 10 + Python 3.5 is used.

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 2, 2015

Thanks. Hopefully I should have this figured out in time for PyInstaller 3.1.

@codewarrior0 codewarrior0 changed the title from Warnings during freeze when using Python 3.5 to Universal CRT not found building with Py3.5 on Win10 Oct 4, 2015

@codewarrior0

This comment has been minimized.

Member

codewarrior0 commented Oct 8, 2015

I found out that there is, in fact, some magic involved.

On Windows 10, the DLLs api-ms-win-crt-*.dll do not actually exist. They are magic words that when passed to the LoadLibrary function, will load a completely different DLL as part of what Windows calls an API Set. This completely different DLL seems to be in all cases ucrtbase.dll.

However, on Windows Vista/7/8, those DLLs do exist. They export functions using the "forwarded exports" feature, where the DLL is redirecting those exports to a different DLL, which is again ucrtbase.dll. So on versions of Windows <10, simply including ucrtbase.dll by itself is not sufficient, and the other DLLs are needed. But when building on Windows 10, we cannot automatically include them because they do not exist, so the only thing to do is to install the VC 2015 runtime on the Vista/7/8 machine.

The runtime is also made available via Windows Update and should be installed on any machine that has automatic updates running.

@cy2k

This comment has been minimized.

cy2k commented Sep 8, 2017

The solution to this problem is to install "Visual C++ Redistributable for Visual Studio 2015."

Sorry, just to be clear, do you mean that the developer that is packaging the code needs to have this installed, or that the end user of the application will need to have this installed in order to run it?

@AcidWeb

This comment has been minimized.

AcidWeb commented Sep 8, 2017

End user.

@cy2k

This comment has been minimized.

cy2k commented Sep 8, 2017

End user.

Thx, @AcidWeb

@xoviat , personally, I don't consider that a solution. I thought the whole I idea of something like pyinstaller is that it's self-contained? Needing the end user to install other dependencies is problematic, in my opinion.

Not trying to be combative, I'm just trying to think about the end user perspective.

@ghost

This comment has been minimized.

ghost commented Sep 8, 2017

Well I don't see a way forward on this issue. PyInstaller cannot bundle these libraries because of MS licensing restrictions and they are required to run on Windows < 10.

@DavidRueter

This comment has been minimized.

DavidRueter commented Sep 8, 2017

@MitchellBerry

This comment has been minimized.

MitchellBerry commented Sep 9, 2017

@htgoebel

This comment has been minimized.

Member

htgoebel commented Sep 9, 2017

Guys, sorry to say, but this "discussion" is not suitable for solving the issue. Please stop offending and suggest what we as PyInstaller team can do to improve the situation.

To repeat my questions (in a bit different way):

  • In which cases is it necessary to install the VSCRT along with an application? For which Python versions? For which Windows versions? Always or only is some extensions are used? Or in different cases for different version of Python?
  • Should PyInstaller collect additional dlls? Is this feasible?
  • Should PyInstaller issue some warning?
  • Should we add a note to the documentation or the Changelog? Please provide a scratch.

Side note: I don't care about any licensing issues here. It's the developers responsibility to verify he/she is allowed to (re-) distribute the software he/she is bundling. He/she has to verify this anyway, so the VSCRT is just one more component. (Although we can help by adding a note to the documentation or issuing a warning.) So it would be okay if PyInstaller collects and VSCRT dlls and adds them to the bundle – if this solves the problem.

@AcidWeb

This comment has been minimized.

AcidWeb commented Sep 9, 2017

@htgoebel main problem there is a fact that there is no universal solution for this problem. The solution varies from project to project and that caused this lengthy discussion and multiple false positive "It works!" posts.

Answering your first question - All binaries prepared by PyInstaller that use Python 3.5+ and Windows as build environment are more or less affected by this problem.

If you use Python 3.5+ most error resistant "solution" is distribute standalone VSCRT installer with your application. But that make PyInstaller output not really stantdalone.

I don't have idea how to really fix this issue.

@MitchellBerry

This comment has been minimized.

MitchellBerry commented Sep 9, 2017

@htgoebel

This comment has been minimized.

Member

htgoebel commented Sep 12, 2017

@AcidWeb Well, what is your advice to developers then? Abstain form Windows? (SCNR). Looking at the mentions of this ticket in other projects and at the duplicates within this project, we need to at least document a solution.

To my understanding #1566 (comment), #1566 (comment) and #1566 (comment) are the essence of this issue. Right?

So I see several possible solutions:

  1. Instruct the developers to either build on Windows 7 or include one of the Redistributable installers into their app's installer. [Edit:] This is Microsofts recommended way, see Distributing Software that uses the Universal CRT, number 2 and 3.
  2. Instruct the developers to install the Windows Software Development Kit (SDK) for Windows 10 and make PyInstaller bundle the required DLLs. [Edit: Added link][Edit:] See Distributing Software that uses the Universal CRT, number 6. According to #1566 (comment) this will add 300KB to the bundled application.

Are these proper solutions? Which one should we implement?

@AcidWeb

This comment has been minimized.

AcidWeb commented Sep 12, 2017

Somebody need to do a comprehensive check if binaries created on Windows 10 + Windows SDK really create binaries that work on all currently used Windows releases. I'm not 100% sure that DLLs taken from SDK will work correctly on older Windows versions.

Otherwise there should be big red note in documentation that using Windows 10 as build environment might not be good idea if somebody need backward compatibility. And inform that in this case VSCRT should be distributed with application.

OS X binaries created by PyInstaller are already affected by the same type of problem for long time and apparently it's time for Windows too :-)

@htgoebel

This comment has been minimized.

Member

htgoebel commented Sep 12, 2017

I'm not 100% sure that DLLs taken from SDK will work correctly on older Windows versions.

Neither am I, but I'd trust Microsofts statement here. It' better then what we have now (which is: nothing).

htgoebel added a commit that referenced this issue Sep 12, 2017

htgoebel added a commit that referenced this issue Sep 12, 2017

@htgoebel

This comment has been minimized.

Member

htgoebel commented Sep 12, 2017

Please comment on #2821

htgoebel added a commit that referenced this issue Sep 16, 2017

htgoebel added a commit that referenced this issue Sep 16, 2017

@bw4sz

This comment has been minimized.

bw4sz commented Sep 26, 2017

Hi everyone, can someone make this solution a bit more tangible. I am experiencing this issue, downloaded the developer kit, and show pyinstaller the --paths to the api-ms-*dlls, but can't get them included.

Stack overflow question is here: https://stackoverflow.com/questions/46416221/pyinstaller-distributing-opencv-from-windows-10-to-windows-10-missing-ucrt-dll

I'll then make a pull request to add a bit to the docs once I have it figured out.

@ThatWeirdAndrew

This comment has been minimized.

ThatWeirdAndrew commented Oct 9, 2017

maybe this will help #2402

@slw07g

This comment has been minimized.

slw07g commented Nov 8, 2017

I had the DLLs on my Windows 10 install...

I specified --path C:\Windows\WinSxS\x86_microsoft-windows-m..namespace-downlevel_31bf3856ad364e35_10.0.15063.0_none_7c5a1866018b960d when I ran pyinstaller, and it worked smoothly.

@JohnCorkren

This comment was marked as resolved.

JohnCorkren commented Jan 11, 2018

Starting with Windows 10 isn't the Universal CRT a part of the operating system? In prior versions of Windows the files were included with Visual C++ redistributable packages.
For Windows 64-bit: Visual C++ Redistributable for Visual Studio 2017 (64-bit)
For Windows 32-bit: Visual C++ Redistributable for Visual Studio 2017 (32-bit)
It could explain why you would see the errors when compiling but the application still runs fine.

Just my guess.............

@htgoebel

This comment has been minimized.

Member

htgoebel commented Jan 12, 2018

@JohnCorkren Please do nor irritate people by guesses. Read all of the issue texts and the manual.

@pyinstaller pyinstaller locked as resolved and limited conversation to collaborators Jan 12, 2018

cowo78 pushed a commit to cowo78/pyinstaller that referenced this issue Dec 7, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.