qt5 5.5.1 breaks PyQt5 (QtDBus)

[The-Compiler]

Since the Qt5 update to 5.5.1, it seems PyQt5 is broken.

When doing import PyQt5.QtGui, I get:

ImportError: dlopen(.../lib/python3.5/site-packages/PyQt5/QtGui.so, 2): Library not loaded: /usr/local/opt/qt5/lib/QtDBus.framework/Versions/5/QtDBus
  Referenced from: .../lib/python3.5/site-packages/PyQt5/QtGui.so
  Reason: image not found

I'd guess this is because of 0c6557f?

/cc @UniqMartin @MikeMcQuaid

[UniqMartin]

Yes, this is quite likely because of the mentioned commit. Can you rebuild pyqt5 (or build it from source, if you used the bottle) and see if that fixes the error (it should)? Thanks!

I'll also have a look. Maybe that's not the only formula that needs a revision bump.

[The-Compiler]

The build seems to fail:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -headerpad_max_install_names -undefined dynamic_lookup -install_name /usr/local/Cellar/pyqt5/5.5_1/lib/python3.5/site-packages/PyQt5
/QtGui.so -framework QtPrintSupport -framework QtDBus -framework QtWidgets -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=10.10 -Wl,-rpat
h,/usr/local/Cellar/qt5/5.5.1/lib -single_module -dynamiclib -o libQtGui.dylib sipQtGuicmodule.o sipQtGuiQAbstractOpenGLFunctions.o sipQtGuiQAbstractTextDocumentLayout.o sipQtGuiQAbstractTextDocumentLayoutPaintContext.o sipQtGu
iQAbstractTextDocumentLayoutSelection.o sipQtGuiQActionEvent.o sipQtGuiQBackingStore.o sipQtGuiQBitmap.o sipQtGuiQBrush.o sipQtGuiQClipboard.o sipQtGuiQCloseEvent.o sipQtGuiQColor.o sipQtGuiQConicalGradient.o sipQtGuiQContextMe
nuEvent.o sipQtGuiQCursor.o sipQtGuiQDesktopServices.o sipQtGuiQDoubleValidator.o sipQtGuiQDrag.o sipQtGuiQDragEnterEvent.o sipQtGuiQDragLeaveEvent.o sipQtGuiQDragMoveEvent.o sipQtGuiQDropEvent.o sipQtGuiQEnterEvent.o sipQtGuiQ
ExposeEvent.o sipQtGuiQFileOpenEvent.o sipQtGuiQFocusEvent.o sipQtGuiQFont.o sipQtGuiQFontDatabase.o sipQtGuiQFontInfo.o sipQtGuiQFontMetrics.o sipQtGuiQFontMetricsF.o sipQtGuiQGlyphRun.o sipQtGuiQGlyphRunGlyphRunFlags.o sipQtG
uiQGradient.o sipQtGuiQGuiApplication.o sipQtGuiQHelpEvent.o sipQtGuiQHideEvent.o sipQtGuiQHoverEvent.o sipQtGuiQIcon.o sipQtGuiQIconDragEvent.o sipQtGuiQIconEngine.o sipQtGuiQIconEngineAvailableSizesArgument.o sipQtGuiQImage.o
 sipQtGuiQImageIOHandler.o sipQtGuiQImageIOHandlerTransformations.o sipQtGuiQImageReader.o sipQtGuiQImageWriter.o sipQtGuiQInputEvent.o sipQtGuiQInputMethod.o sipQtGuiQInputMethodEvent.o sipQtGuiQInputMethodEventAttribute.o sip
QtGuiQInputMethodQueryEvent.o sipQtGuiQIntValidator.o sipQtGuiQKeyEvent.o sipQtGuiQKeySequence.o sipQtGuiQLinearGradient.o sipQtGuiQList0100QFontDatabaseWritingSystem.o sipQtGuiQList0100QGlyphRun.o sipQtGuiQList0100QInputMethod
EventAttribute.o sipQtGuiQList0100QKeySequence.o sipQtGuiQList0100QOpenGLDebugMessage.o sipQtGuiQList0100QPolygonF.o sipQtGuiQList0100QSize.o sipQtGuiQList0100QTextBlock.o sipQtGuiQList0100QTextLayoutFormatRange.o sipQtGuiQList
0100QTextOptionTab.o sipQtGuiQList0100QTouchEventTouchPoint.o sipQtGuiQList0101QOpenGLContext.o sipQtGuiQList0101QOpenGLShader.o sipQtGuiQList0101QScreen.o sipQtGuiQList0101QStandardItem.o sipQtGuiQList0101QTextFrame.o sipQtGui
QList0101QWindow.o sipQtGuiQList0111QTouchDevice.o sipQtGuiQMatrix2x2.o sipQtGuiQMatrix2x3.o sipQtGuiQMatrix2x4.o sipQtGuiQMatrix3x2.o sipQtGuiQMatrix3x3.o sipQtGuiQMatrix3x4.o sipQtGuiQMatrix4x2.o sipQtGuiQMatrix4x3.o sipQtGui
QMatrix4x4.o sipQtGuiQMouseEvent.o sipQtGuiQMoveEvent.o sipQtGuiQMovie.o sipQtGuiQNativeGestureEvent.o sipQtGuiQOffscreenSurface.o sipQtGuiQOpenGLBuffer.o sipQtGuiQOpenGLBufferRangeAccessFlags.o sipQtGuiQOpenGLContext.o sipQtGu
iQOpenGLContextGroup.o sipQtGuiQOpenGLDebugLogger.o sipQtGuiQOpenGLDebugMessage.o sipQtGuiQOpenGLDebugMessageSeverities.o sipQtGuiQOpenGLDebugMessageSources.o sipQtGuiQOpenGLDebugMessageTypes.o sipQtGuiQOpenGLFramebufferObject.
o sipQtGuiQOpenGLFramebufferObjectFormat.o sipQtGuiQOpenGLPaintDevice.o sipQtGuiQOpenGLPixelTransferOptions.o sipQtGuiQOpenGLShader.o sipQtGuiQOpenGLShaderProgram.o sipQtGuiQOpenGLShaderShaderType.o sipQtGuiQOpenGLTexture.o sip
QtGuiQOpenGLTextureFeatures.o sipQtGuiQOpenGLTimeMonitor.o sipQtGuiQOpenGLTimerQuery.o sipQtGuiQOpenGLVersionProfile.o sipQtGuiQOpenGLVertexArrayObject.o sipQtGuiQOpenGLVertexArrayObjectBinder.o sipQtGuiQOpenGLWindow.o sipQtGui
QPagedPaintDevice.o sipQtGuiQPagedPaintDeviceMargins.o sipQtGuiQPageLayout.o sipQtGuiQPageSize.o sipQtGuiQPaintDevice.o sipQtGuiQPaintDeviceWindow.o sipQtGuiQPaintEngine.o sipQtGuiQPaintEngineDirtyFlags.o sipQtGuiQPaintEnginePa
intEngineFeatures.o sipQtGuiQPaintEngineState.o sipQtGuiQPainter.o sipQtGuiQPainterPath.o sipQtGuiQPainterPathElement.o sipQtGuiQPainterPathStroker.o sipQtGuiQPainterPixmapFragment.o sipQtGuiQPainterPixmapFragmentHints.o sipQtG
uiQPainterRenderHints.o sipQtGuiQPaintEvent.o sipQtGuiQPair0100QOpenGLTextureFilter0100QOpenGLTextureFilter.o sipQtGuiQPair22002200.o sipQtGuiQPalette.o sipQtGuiQPdfWriter.o sipQtGuiQPen.o sipQtGuiQPicture.o sipQtGuiQPictureIO.
o sipQtGuiQPixelFormat.o sipQtGuiQPixmap.o sipQtGuiQPixmapCache.o sipQtGuiQPixmapCacheKey.o sipQtGuiQPlatformSurfaceEvent.o sipQtGuiQPolygon.o sipQtGuiQPolygonF.o sipQtGuiQQuaternion.o sipQtGuiQRadialGradient.o sipQtGuiQRasterW
indow.o sipQtGuiQRawFont.o sipQtGuiQRawFontLayoutFlags.o sipQtGuiQRegExpValidator.o sipQtGuiQRegion.o sipQtGuiQRegularExpressionValidator.o sipQtGuiQResizeEvent.o sipQtGuiQScreen.o sipQtGuiQScrollEvent.o sipQtGuiQScrollPrepareE
vent.o sipQtGuiQSessionManager.o sipQtGuiQSet0100QByteArray.o sipQtGuiQShortcutEvent.o sipQtGuiQShowEvent.o sipQtGuiQStandardItem.o sipQtGuiQStandardItemModel.o sipQtGuiQStaticText.o sipQtGuiQStatusTipEvent.o sipQtGuiQStyleHint
s.o sipQtGuiQSurface.o sipQtGuiQSurfaceFormat.o sipQtGuiQSurfaceFormatFormatOptions.o sipQtGuiQSyntaxHighlighter.o sipQtGuiQt.o sipQtGuiQTabletEvent.o sipQtGuiQTextBlock.o sipQtGuiQTextBlockFormat.o sipQtGuiQTextBlockGroup.o si
pQtGuiQTextBlockiterator.o sipQtGuiQTextBlockUserData.o sipQtGuiQTextCharFormat.o sipQtGuiQTextCursor.o sipQtGuiQTextDocument.o sipQtGuiQTextDocumentFindFlags.o sipQtGuiQTextDocumentFragment.o sipQtGuiQTextDocumentWriter.o sipQ
tGuiQTextFormat.o sipQtGuiQTextFormatPageBreakFlags.o sipQtGuiQTextFragment.o sipQtGuiQTextFrame.o sipQtGuiQTextFrameFormat.o sipQtGuiQTextFrameiterator.o sipQtGuiQTextImageFormat.o sipQtGuiQTextInlineObject.o sipQtGuiQTextItem
.o sipQtGuiQTextItemRenderFlags.o sipQtGuiQTextLayout.o sipQtGuiQTextLayoutFormatRange.o sipQtGuiQTextLength.o sipQtGuiQTextLine.o sipQtGuiQTextList.o sipQtGuiQTextListFormat.o sipQtGuiQTextObject.o sipQtGuiQTextObjectInterface
.o sipQtGuiQTextOption.o sipQtGuiQTextOptionFlags.o sipQtGuiQTextOptionTab.o sipQtGuiQTextTable.o sipQtGuiQTextTableCell.o sipQtGuiQTextTableCellFormat.o sipQtGuiQTextTableFormat.o sipQtGuiQTouchDevice.o sipQtGuiQTouchDeviceCap
abilities.o sipQtGuiQTouchEvent.o sipQtGuiQTouchEventTouchPoint.o sipQtGuiQTouchEventTouchPointInfoFlags.o sipQtGuiQTransform.o sipQtGuiQValidator.o sipQtGuiQVector0100QAbstractTextDocumentLayoutSelection.o sipQtGuiQVector0100Q
Line.o sipQtGuiQVector0100QLineF.o sipQtGuiQVector0100QPoint.o sipQtGuiQVector0100QRect.o sipQtGuiQVector0100QRectF.o sipQtGuiQVector0100QTextFormat.o sipQtGuiQVector0100QTextLayoutFormatRange.o sipQtGuiQVector0100QTextLength.o
 sipQtGuiQVector0600QPair24000100QColor.o sipQtGuiQVector1900.o sipQtGuiQVector2200.o sipQtGuiQVector2400.o sipQtGuiQVector2D.o sipQtGuiQVector3900.o sipQtGuiQVector3D.o sipQtGuiQVector4D.o sipQtGuiQWhatsThisClickedEvent.o sipQ
tGuiQWheelEvent.o sipQtGuiQWindow.o sipQtGuiQWindowStateChangeEvent.o qpyopengl_add_constants.o qpyopengl_array_convertors.o qpyopengl_attribute_array.o qpyopengl_data_cache.o qpyopengl_init.o qpyopengl_uniform_value_array.o qp
yopengl_value_array.o qpyopengl_version_functions.o  -F/usr/local/Cellar/qt5/5.5.1/lib -framework QtGui -framework QtCore -framework DiskArbitration -framework IOKit -framework OpenGL -framework AGL  
ld: framework not found QtDBus
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [libQtGui.dylib] Error 1
make: *** [sub-QtGui-make_first-ordered] Error 2

[DomT4]

I wonder why the CI didn't break on testing PyQt5 for the Qt5 bump in this case 😕.

[UniqMartin]

I wonder why the CI didn't break on testing PyQt5 for the Qt5 bump in this case.

It only runs the tests of dependent formulae to check that nothing breaks, right? And it does so by using whatever is in a bottle of a formula. For this specific case, it only tries to import PyQt5 and that does not pull in any of the submodules (or whatever that is called in Python land). To detect the breakage, the test would have had to do import PyQt5.QtGui.

Seems like pyqt5 has a hard dependency on the QtDBus module, but only on OS X. The reasons for this are not obvious to me and if I patch out the dependency in configure.py, everything compiles fine and a seemingly working pyqt5 results.

Possible solutions to the breakage:

• Patch out the QtDBus dependency in pyqt5 unless qt5 was explicitly built --with-d-bus. Notify upstream, find out with their help how to properly fix this (configure.py should probably react to the absence of the QtDBus module), and hopefully have an officially sanctioned fix at some later time.
• Re-enable the QtDBus module unconditionally and be done. (I tried to be smart by disabling stuff that seemed to be useless on OS X, but that obviously wasn't that smart.)

I'd favor the first solution if there's someone willing to deal with upstream. (Not me, I'm just not a Python guy.) I'd gladly forward all the knowledge I gathered while debugging this. Otherwise, let's just revert the offending commit. Opinions?

[UniqMartin]

@The-Compiler Thanks for reporting this and for attempting to build from source! If your main priority is to get back to a working setup, you could:

• Either reinstall qt5 with D-Bus support: brew remove qt5 && brew install qt5 --with-d-bus.
• Or merge my pyqt5-no-dbus branch: master...UniqMartin:pyqt5-no-dbus. Building pyqt5 from source should succeed then. (Only a temporary solution until we figure out how to best fix this.)

[The-Compiler]

Building Qt with --with-d-bus worked - it also only took 6 hours on my poor old Mac Mini ⌛. Unfortunately this isn't an option on Travis, where my builds are currently failing 😉

I notified upstream - judging from my past experience, the answer will probably be "fixed in tonight's snapshot" in a few hours 😉

Is there an easy way I can install from that branch with brew on Travis? I think building PyQt by hand should work in the maximum time I get from Travis, so that'd be a good workaround for the time being.

[The-Compiler]

I just got an answer from Phil, the guy behind PyQt:

Already fixed. It's there as a workaround for OS X rpath related bugs in Qt v5.5.x.

Seems like the proper fix is:

--- PyQt-gpl-5.5/configure.py   2015-07-17 13:45:40.000000000 +0200
+++ PyQt-gpl-5.5.1-snapshot-13f9ece29d02/configure.py   2015-10-19 03:31:39.000000000 +0200
@@ -2478,9 +2504,25 @@
         pro_lines.append('LIBS += %s' % libs)

     if not target_config.static:
-        # Make sure these frameworks are already loaded by the time the
-        # libqcocoa.dylib plugin gets loaded.
-        extra_lflags = 'QMAKE_LFLAGS += "-framework QtPrintSupport -framework QtDBus -framework QtWidgets"\n        ' if mname == 'QtGui' else ''
+        # For Qt v5.5 and later, Make sure these frameworks are already loaded
+        # by the time the libqcocoa.dylib plugin gets loaded.  This problem is
+        # due to be fixed in Qt v5.6.
+        extra_lflags = ''
+
+        if mname == 'QtGui':
+            # Note that this workaround is flawed because it looks at the PyQt
+            # configuration rather than the Qt configuration.  It will fail if
+            # the user is building a PyQt without the QtDBus module against a
+            # Qt with the QtDBus library.  However it will be fine for the
+            # common case where the PyQt configuration reflects the Qt
+            # configuration.
+            fwks = []
+            for m in ('QtPrintSupport', 'QtDBus', 'QtWidgets'):
+                if m in target_config.pyqt_modules:
+                    fwks.append('-framework ' + m)
+
+            if len(fwks) != 0:
+                extra_lflags = 'QMAKE_LFLAGS += "%s"\n        ' % ' '.join(fwks)

         shared = '''
 win32 {

I haven't tested this yet, and there are loads of other changes to configure.py between the release and snapshot. @UniqMartin, any chance you could prepare a formula with that? Then I could test it - I'm not experienced with Homebrew/Ruby myself.

Patching the pyqt5.rb formula with @UniqMartin's workaround and building with --build-from-source works for now (and only builds for ~13 minutes):

print("Patching PyQt formula...")
brew_path = subprocess.check_output(['brew', '--prefix']).strip()
pyqt_file = os.path.join(brew_path, 'Library', 'Formula', 'pyqt5.rb')
os.remove(pyqt_file)
urllib.urlretrieve(
    'https://raw.githubusercontent.com/UniqMartin/homebrew/0f78553c1cfc26963ea0681374e50b378207f875/Library/Formula/pyqt5.rb',
    pyqt_file)

[MikeMcQuaid]

Re-enable the QtDBus module unconditionally and be done. (I tried to be smart by disabling stuff that seemed to be useless on OS X, but that obviously wasn't that smart.)

This seems to make most sense to me.

[DomT4]

My worry here is that if there's silent breakage the CI didn't pick up on there may well be elsewhere in similar situations. I'd be tempted to agree about reenabling QtDBus, sadly.

[UniqMartin]

@The-Compiler Thanks for contacting upstream! It's great to have a proper fix as quickly as this. The change you highlighted is the one we need and it applies cleanly, so I prepared a new branch for you to test: the branch, just the formula. Will create a PR once I have your confirmation that it works.

@MikeMcQuaid @DomT4 Since we have a proper upstream fix, I strongly prefer that. We can still revert the qt5 change later, if the backlash from my change turns out to be too big. Plus it is a lot less painful to rebuild pyqt5 than to rebuild qt5.

@DomT4 I have a proposal about how the CI could detect those kinds of breakages automatically, though that won't be an immediate or short-term solution. I'll make sure to CC you once I've posted the proposal as a new issue.

[The-Compiler]

Travis is building here, let's see if this works: https://travis-ci.org/The-Compiler/qutebrowser/jobs/86205210

You didn't re-add the import PyQt5.QtGui test though - I think some others would also make sense to be added, like QtWidgets or QtWebKit.

[The-Compiler]

  py35: commands succeeded
  congratulations :)

So yup, seems to work fine for me 👍

[UniqMartin]

You didn't re-add the import PyQt5.QtGui test though - I think some others would also make sense to be added, like QtWidgets or QtWebKit.

This wasn't intentional. I agree and will make sure to re-add these before submitting the PR.

[UniqMartin]

Submitted PR #45145. Travis is already happy with it. 😉

[davelab6]

Awesome work@ @DomT4 you might like to know that I just hit this as I'm writing install docs for @trufont :)

[DomT4]

you might like to know that I just hit this as I'm writing install docs for @trufont :)

Using PyQt5 or something else?

[davelab6]

Yep. It's a fontforge replacement

[DomT4]

Ah, interesting! Looks like it's based off a Qt backend, which will be quite a departure from the land of Quartz. If you hit the problem seen in PyQt5 anywhere else let us know and we'll try and hunt that down as well.