New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

This application failed to start because it could not find or load the Qt platform plugin "cocoa" (patches not working) #3175

Closed
jkempinger-dv opened this Issue Jan 3, 2018 · 28 comments

Comments

Projects
None yet
4 participants
@jkempinger-dv

jkempinger-dv commented Jan 3, 2018

Hi, I have been using PyInstaller with python 3.5.2 and PyQt5/Qt5 5.7 for some time now and it has been working great. This week I needed to update my build environment to python 3.6.4 and PyQt5/Qt5 5.9.2. Now when I run my packaged application I see the following (paths truncated with ...):

objc[22702]: Class RunLoopModeTracker is implemented in both .../Contents/MacOS/QtCore (0x10a9c6fe0) and .../Qt5.9.2/5.9.2/clang_64/lib/QtCore.framework/Versions/5/QtCore (0x118486fe0). One of the two will be used. Which one is undefined.
objc[22702]: Class NotificationReceiver is implemented in both .../Contents/MacOS/QtWidgets (0x10bfc0218) and .../Qt5.9.2/5.9.2/clang_64/lib/QtWidgets.framework/Versions/5/QtWidgets (0x117969218). One of the two will be used. Which one is undefined.
This application failed to start because it could not find or load the Qt platform plugin "cocoa"
in "".

Reinstalling the application may fix this problem.

I was using PyInstaller 3.3.1 but also see the same issues with the development version (installed today).

I added print(QApplication.libraryPaths()) in my application code (before initializing the QApplication) and it returns an empty list.

If I apply 082078e, referenced in issue #2924, I see the following:

objc[22702]: Class RunLoopModeTracker is implemented in both .../Contents/MacOS/QtCore (0x10a9c6fe0) and .../Qt5.9.2/5.9.2/clang_64/lib/QtCore.framework/Versions/5/QtCore (0x118486fe0). One of the two will be used. Which one is undefined.
objc[22702]: Class NotificationReceiver is implemented in both .../Contents/MacOS/QtWidgets (0x10bfc0218) and .../Qt5.9.2/5.9.2/clang_64/lib/QtWidgets.framework/Versions/5/QtWidgets (0x117969218). One of the two will be used. Which one is undefined.
['.../Contents/MacOS/qt5_plugins']
This application failed to start because it could not find or load the Qt platform plugin "cocoa"
in "".

Reinstalling the application may fix this problem.

Note the print statement I added is now showing a plugins directory, but it's incorrect. PyInstaller appears to have packaged the plugins in .../Contents/MacOS/PyQt5/Qt/plugins.

If I modify pyi_rth_qt5plugins.py to use this path the application runs (though I still see the double implementation warnings):

objc[22702]: Class RunLoopModeTracker is implemented in both .../Contents/MacOS/QtCore (0x10a9c6fe0) and .../Qt5.9.2/5.9.2/clang_64/lib/QtCore.framework/Versions/5/QtCore (0x118486fe0). One of the two will be used. Which one is undefined.
objc[22702]: Class NotificationReceiver is implemented in both .../Contents/MacOS/QtWidgets (0x10bfc0218) and .../Qt5.9.2/5.9.2/clang_64/lib/QtWidgets.framework/Versions/5/QtWidgets (0x117969218). One of the two will be used. Which one is undefined.
['.../Contents/MacOS/PyQt5/Qt/plugins']

I have also tried including a qt.conf file in .../Contents/Resources but it doesn't seem to be read by Qt as the path (from my print statement) never changes regardless of what I set for 'Plugins' in qt.conf.

The other patch from issue #2924 seems to already be applied in the development version of PyInstaller. I still don't have a solid grasp on the build process with PyInstaller, and I'm not sure what to try next. I'd appreciate any help! Thanks!

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 3, 2018

Thanks for a thorough description and investigation of the problem. Unfortunately, PyInstaller (attempts to) support only the latest version of a given library, which is PyQt 5.9 in this case. Would it be possible for you to upgrade? I'm currently working on fixing a few PyQt5 bugs with PyInstaller, but lack a Mac on which to test. Your help in testing would be appreciated, if you can.

@jkempinger-dv

This comment has been minimized.

jkempinger-dv commented Jan 3, 2018

@bjones1 I am actually currently using PyQt5 5.9.2. I was using 5.7 previously and wasn't seeing these issues, but have been needing to upgrade my build environment for a while now and finally took the plunge.

I believe the older versions of PyQt5 were using the qt5_plugins path and the newer version(s) are using PyQt5/Qt/plugins, though I'm not entirely sure if the paths are due to PyQt, Qt, or PyInstaller?

I would be happy to help with testing! Any ideas on the double implementation warnings? If I move my build folder (I am building mostly everything from source) to another directory I no longer see them, but I suspect there is either something wrong with my build environment or something that's changed in PyInstaller.

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 3, 2018

Oops, I didn't read carefully enough re: the PyQt5 version you're currently using. The changes are all based on the new wheel layout of PyQt5. How did you install -- via pip, or brew/macports/whatever?

@jkempinger-dv

This comment has been minimized.

jkempinger-dv commented Jan 3, 2018

I installed PyQt from source to a local user-owned build directory using PREFIX. We had issues in the past installing PyQt from pip or homebrew and switched to compiling it myself.

I'm still trying to get Qt to read qt.conf for the plugin path, as it would provide more control over the build process, but it seems to only read the path that PyInstaller sets with pyi_rth_qt5plugins.py for loading the qcocoa plugin.

Here's some paths that Qt reports when running in the built version:

  • os.environ.get('QT_PLUGIN_PATH')) is None
  • QApplication.libraryPaths() is a single-element list set from pyi_rth_qt5plugins.py before instantiating a QApplication
  • QApplication.libraryPaths() after instantiating a QApplication contains the first element from before and also contains the path to the executable ('.../Contents/MacOS'), as expected from the Qt docs
  • QLibraryInfo.location(QLibraryInfo.PluginsPath), before or after instantiation is set to the path specified in qt.conf (in '.../Contents/Resources')

With the exception of QLibraryInfo, nothing in the qt.conf file seems to be read, but it appears Qt isn't searching QLibraryInfo.location(QLibraryInfo.PluginsPath) for plugins (e.g. qcocoa) before instantiation, so qt.conf changes nothing...

So it looks like Qt is only reading QApplication.libraryPaths() to find the qcocoa plugin and the only place that is set is in pyi_rth_qt5plugins.py (with QCoreApplication.setLibraryPaths([os.path.abspath(d)])) and the required path differs between older and newer PyQt versions.

EDIT: Would setting the library paths to both qt5_plugins and PyQt5/Qt/plugins be a feasible solution? Presumably Qt would only search the path that exists.

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 3, 2018

My goal is PyQt5 support is to have PyInstaller's setup mirror Python's as closely as possible. I've made a few tweaks at https://github.com/bjones1/pyinstaller/tree/pyqt5_testing, but plan on a bit more. So, I'll eliminate the call to QCoreApplication.setLibraryPaths([os.path.abspath(d)]) -- I thought I'd already disabled that hook for PyQt5, but evidently not. Would you mind removing this from the run-time hook to see if that helps?

In general, copying all of the PyQt5/ directory to build/appname/PyQt5 should fix all problems in this layout. PyInstaller will try to copy a subset (only those files actually needed), but this still needs fixing.

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 3, 2018

For more (well, too much) info, see #2991 (comment) and 675d66a#commitcomment-26451340...

@tallforasmurf

This comment has been minimized.

Contributor

tallforasmurf commented Jan 3, 2018

In the past I've spent many frustrating hours trying to "deploy" PyQt apps, so my hat's off to you for trying. I have 5.9 installed on MacOS, so if you have some specific test you want run there, I could do it.

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 3, 2018

@tallforasmurf - great, thanks. I'd like to work a bit more on my branch, then I'll definitely ask for some help and tests. Are you set up to run the PyInstaller test (pytest)?

@jkempinger-dv

This comment has been minimized.

jkempinger-dv commented Jan 3, 2018

@bjones1 The pyi_rth_qt5plugins.py hook is not in the current version of PyInstaller, but without that hook the search path is just an empty list, and I get This application failed to start because it could not find or load the Qt platform plugin "cocoa" in "". on launch. In issue #2924 one of the listed fixes is to re-enable this hook, which presumably would work for older PyQt5 versions that still use the qt5_plugins path, but does not work for me with PyQt 5.9.2 which uses the PyQt5/Qt/plugins path.

I tried copying everything in my PyQt5 build directory (build/lib/python3.6/site-packages/PyQt5) into the PyQt5 directory that resides next to the PyInstaller built application, but without the modified pyi_rth_qt5plugins.py hook it still can't find the cocoa lib. I also tried placing those files directly adjacent to the executable, but I get the same error.

I think that PyInstaller is copying all the files it needs, but something isn't being set up with Qt to tell it where to search for those libraries. The call to QCoreApplication.setLibraryPaths([os.path.abspath(d)]) takes care of this, but only if the analogous path for the version of PyQt5 is being used.

@tallforasmurf

This comment has been minimized.

Contributor

tallforasmurf commented Jan 3, 2018

No, I've never run the PyInstaller tests. If I clone the dev branch, is that something builtin that I would find in it?

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 3, 2018

OK, thanks for clarifying. Hmmm. My big question is, on the Mac, how does Python itself find all the necessary files? I'd like to duplicate that if possible. I can't find a qt.conf in the Mac wheel -- is there one in your install? Failing this, what's the correct path to put in the call to QCoreApplication.setLibraryPaths([os.path.abspath(d)])?

My other concern: I want to first and foremost support the pip-installed version, since that's what the Travis tests run on. Would you be able to use the pip version, to make sure these changes work there as well?

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 3, 2018

@tallforasmurf, yes. Something like

git clone https://github.com/pyinstaller/pyinstaller.git
cd pyinstaller/tests
python3 -m pip install -U -r requirements-tools.txt
python3 -m pip install -U -r requirements-developer.txt
pytest -k test_PyQt5_QWebEngine[onedir]

would run one of the harder tests (QtWebEngine functionality) on the Mac. Though you will need to delete a line first.

@jkempinger-dv

This comment has been minimized.

jkempinger-dv commented Jan 3, 2018

Hmm, that is a good question... When I run my application directly in python I see QApplication.libraryPaths() is set correctly to Qt's plugins folder (...build/Qt5.9.2/5.9.2/clang_64/plugins) in my build directory. I don't think I've manually set this anywhere though. From what I can tell Qt defaults to 'plugins', relative to QCoreApplication::applicationDirPath() and I am not using a qt.conf file to override anything. For me this seems to be ...build/bin/. I'm not sure where that is set, but it's in my shell's PATH, so maybe it's getting it from there? However, that doesn't explain where it's getting the correct plugins path from. I still am not sure what/who is setting Qt's library path correctly when running at the CLI...

EDIT: I found this note that says that the installation path + /plugins is the default path. So that would explain why it works when running through python at the CLI. That path will not work for a PyInstaller built version since it needs to be portable for other systems... But even so it seems that that path is not set when I run the built version, so presumably either PyInstaller removes it at some point along the way or there's some Qt file that stores that setting that is not being copied to the built application's directory.

@jkempinger-dv

This comment has been minimized.

jkempinger-dv commented Jan 3, 2018

We're using the commercial version of PyQt, so I don't believe we can build it via pip with the licensed headers. But I can try a pip-built PyQt5 just for testing and get back to you.

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 3, 2018

For me, QCoreApplication.libraryPaths() reports paths that are absolute, but somehow follows the location of the plugins folder around. That is, it's more than just QCoreApplication.applicationDirPath(). How does PyQt5 "know" this? This build-in "knowledge" seems to apply to Windows and Linux, but not Mac (unless there's some magic in the wheel version of PyQt5). Specifically, on my side (Windows), with Python:

(venv) E:\pyinstaller\tests>python
Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyQt5.QtCore import QCoreApplication
>>> qa = QCoreApplication([])
>>> print(qa.applicationDirPath())
E:/pyinstaller/venv/Scripts
>>> print(qa.libraryPaths())
['E:/pyinstaller/venv/lib/site-packages/PyQt5/Qt/plugins', 'E:/pyinstaller/venv/Scripts']
>>>

Using PyInstaller:

(venv) E:\pyinstaller\tests>C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths\test_PyQt5_libraryPaths.exe
[4748] PyInstaller Bootloader 3.x
[4748] LOADER: executable is C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths\test_PyQt5_libraryPaths.exe
[4748] LOADER: homepath is C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths
[4748] LOADER: _MEIPASS2 is NULL
[4748] LOADER: archivename is C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths\test_PyQt5_libraryPaths.exe
[4748] LOADER: No need to extract files to run; setting extractionpath to homepath
[4748] LOADER: SetDllDirectory(C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths)
[4748] LOADER: Already in the child - running user's code.
[4748] LOADER: Python library: C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths\python36.dll
[4748] LOADER: Loaded functions from Python library.
[4748] LOADER: Manipulating environment (sys.path, sys.prefix)
[4748] LOADER: Pre-init sys.path is C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths\base_library.zip;C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths
[4748] LOADER: sys.prefix is C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths
[4748] LOADER: Setting runtime options
[4748] LOADER: Initializing python
[4748] LOADER: Overriding Python's sys.path
[4748] LOADER: Post-init sys.path is C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths\base_library.zip;C:\Users\bjones\AppData\Local\Temp\pytest-of-bjones\pytest-18\test_PyQt5_libraryPaths_onedir0\dist\test_PyQt5_libraryPaths
[4748] LOADER: Setting sys.argv
[4748] LOADER: setting sys._MEIPASS
[4748] LOADER: importing modules from CArchive
[4748] LOADER: extracted struct
[4748] LOADER: callfunction returned...
[4748] LOADER: extracted pyimod01_os_path
[4748] LOADER: callfunction returned...
[4748] LOADER: extracted pyimod02_archive
[4748] LOADER: callfunction returned...
[4748] LOADER: extracted pyimod03_importers
[4748] LOADER: callfunction returned...
[4748] LOADER: Installing PYZ archive with Python modules.
[4748] LOADER: PYZ archive: out00-PYZ.pyz
[4748] LOADER: Running pyiboot01_bootstrap.py
[4748] LOADER: Running pyi_rth_qt5.py
[4748] LOADER: Running test_PyQt5_libraryPaths.py
C:/Users/bjones/AppData/Local/Temp/pytest-of-bjones/pytest-18/test_PyQt5_libraryPaths_onedir0/dist/test_PyQt5_libraryPaths
['C:/Users/bjones/AppData/Local/Temp/pytest-of-bjones/pytest-18/test_PyQt5_libraryPaths_onedir0/dist/test_PyQt5_libraryPaths/PyQt5/Qt/plugins', 'C:/Users/bjones/AppData/Local/Temp/pytest-of-bjones/pytest-18/test_PyQt5_libraryPaths_onedir0/dist/test_PyQt5_libraryPaths']
[4748] LOADER: OK.
[4748] LOADER: Cleaning up Python interpreter.
@jkempinger-dv

This comment has been minimized.

jkempinger-dv commented Jan 4, 2018

I created a fresh build environment, but pip installed the GPL version this time. We won't be able to distribute our software using this route, but I figured it was worth testing. It's working... I rolled back the changes I made to PyInstaller so it's currently not running pyi_rth_qt5plugins.py anymore, but the QCoreApplication.libraryPaths() are set correctly when running through python and after building with PyInstaller.

So it does seem that something is broken with my build from source... I'm not sure what the pip installer would be doing differently? It looks like there is a way to build a licensed wheel from the commercial version and install with pip. So I'll give that a try and see if that works!

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 4, 2018

Wow. Thanks so much for trying this. I think this is probably work a question on the PyQt mailing list: when built from source, what determines the library path? When built from a wheel, what determines the library path?

By the pip installer, do you mean what does pip do when installing a wheel? If so, a wheel is just a .zip file (you can rename it then inspect it using unzip or whatever). So, when pip installs the wheel, it just unzips the files; there's no patching/post-processing. That is to say, the library path must have already been built into the wheel.

Thanks again for putting some time and energy into investigating this!

@jkempinger-dv

This comment has been minimized.

jkempinger-dv commented Jan 4, 2018

No problem! PyInstaller is an important part of our application's build process, so I'm happy to help where I can!

I was able to install the commercial version of PyQt5 using the wheel that Riverbank provides and everything is working fine. No more errors about finding the platform plugin so I'm able to use stock PyInstaller 3.3.1!

When I get a chance I'll post a message to PyQt's mailing list to see what might be different about the source install. Thanks for all your help, and let me know if there's any more debugging I can do on macOS!

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 17, 2018

@jkempinger-dv and @tallforasmurf, I've got my PyQt5 updates running on Windows and Linux. Are either of yall available to help test and debug these changes on a Mac? (My Mac Travis build is stuck.) If so, please clone my branch and either run pytest -k test_PyQt5 or try running Pyinstaller on this script:

from PyQt5.QtWidgets import QApplication
from PyQt5.QtWebEngineWidgets import QWebEngineView
from PyQt5.QtCore import QUrl
app = QApplication( [] )
view = QWebEngineView()
view.load( QUrl( "http://www.pyinstaller.org" ) )
view.show()
view.page().loadFinished.connect(lambda ok: app.quit())
app.exec_()
@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 17, 2018

FYI, Travis finally ran my Mac build, revealing a few bugs which I just fixed. Be sure to pull the latest when you test...

@ghost

This comment has been minimized.

ghost commented Jan 17, 2018

@bjones1 If you need fast OSX iteration you can use Travis-CI staging. It's probably unethical though.

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 17, 2018

Sigh. It's the Mac story -- I could (probably) run OSX in VirtualBox on my PC, but that's illegal. I could probably run more often on Travis, but that's unethical. I wish developing for an Apple didn't require Apple hardware. It also doesn't help that today, Travis is having a bad MacOS day...

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 17, 2018

On a side note, the way you've integrated testing in Appveyor is really nice -- the "tests" tab output is quite helpful.

@htgoebel

This comment has been minimized.

Member

htgoebel commented Jan 18, 2018

@bjones: No plaintiff, no judge. You can use the machines prepared in the Vagrant file.
For travis-testing you could strip down the .travis.yml to only run the required tests (and e.g. not build the bootloader) and only on OS X.

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 18, 2018

I'm unfamiliar with the Vagrant file -- pointers would be appreciated. I've put a minimal build in the Travis queue and have been waiting for 10+ hours for the build to even start :(

@htgoebel

This comment has been minimized.

Member

htgoebel commented Jan 18, 2018

In short something along these lines:

  • install vagrant
  • cd bootloader
  • export GUI=1 # make the Vm show up
  • vagrant up --no-provision darwin64 # provsion is meant to build the bootloader

For more details see doc/bootloader-building.rst:
HTH.

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 18, 2018

Thanks! I'll take a look.

@bjones1 bjones1 referenced this issue Jan 22, 2018

Closed

PyQt5 fix #3233

@bjones1

This comment has been minimized.

Member

bjones1 commented Jan 31, 2018

I'm closing this, since #3233 should fix it -- see the instructions there.

@bjones1 bjones1 closed this Jan 31, 2018

@htgoebel htgoebel added this to the PyInstaller 3.4 milestone Sep 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment