Switching to Qt 6 by default

[The-Compiler]

(Continuation of #5395 because that was getting rather big)

Current state

On July 22th, 2023, the flip was finally switched to Qt 6 by default. All you need to ensure is what PyQt6 is available, and Qt 6 will be used automatically. If it's not available, you will get a warning shown by qutebrowser.

All information below is considered historical.

The qt6-v2 branch has been merged to master on March 18th, 2023, but with Qt 5 still being the default. See #7628 for details. This issue is now repurposed on what there needs to be done for Qt 6 to be the default Qt wrapper.

The last fixes for the qt6-v2 branch with Qt 6 support went in around May 19th. Multiple core developers (and users) have been using it as a daily driver since far longer (late 2021 / early 2022) and all known crashes are fixed, with all tests passing.

There is still a lot of remaining work, however, on the development side. Due to the big amount of changes (currently 156 commits, 314 files changed, 12,978 additions and 4,094 deletions) it's tricky to integrate this into master without breaking lots of existing open PRs. Thus, we'd like to do another round of PR merging before attempting to do so. The current approach also breaks various tooling, by not properly supporting static analysis (#7128, but that will introduce another big code churn), or by not fixing linting issues yet (#1455).

Bottomline: Feel free to use the qt6-v2 branch, keep in mind it will be rebased from time to time, and it will still take lots of efforts behind the scenes to get it fully integrated.

How to try

mkvenv.py

• Take a backup of your qutebrowser data directory (see :version) unless you don't mind losing cookies and such when down-/upgrading. This isn't needed if you use --temp-basedir below, but if you forget to do so, your data will get upgraded, and as far as I'm aware, there is no way to read it with Qt 5 after that.
• Clone the git repo, e.g. via git clone https://github.com/qutebrowser/qutebrowser.git
• cd qutebrowser
• Check out the branch using git checkout qt6-v2
• Create a new separate environment via python3 scripts/mkvenv.py --venv-dir .venv-qt6 --pyqt-version 6.3
• Set QUTE_QT_WRAPPER=PyQt6 somewhere in the environment
• Run .venv-qt6/bin/qutebrowser --temp-basedir and verify via :version that you're indeed on Qt 6

Archlinux

qutebrowser-qt6-git is available in the AUR.

Windows / macOS

Built as part of the nightly builds on CI. Click on the most recent passing build, then find the qt6 builds under "Artifacts". Note that you will need to be logged in with a GitHub account to download.

Note for macOS: Due to a PyInstaller issue, Chromium's sandboxing is currently disabled for macOS builds with Qt 6. This means that there will be no additional layer of protection in case of Chromium security bugs. Thus, it's advised to not use this build in production. Hopefully, this situation will be resolved before the final 3.0.0 release.

Flatpak

Published in the flathub-beta repo:

Adding the flathub-beta remote and installing from the beta branch can be done through an application manager (Gnome Software, Discover, ...) or via the command-line.

$ flatpak remote-add --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo
$ flatpak install flathub-beta org.qutebrowser.qutebrowser//beta

Remaining issues

• Flickering rectangles with AMD graphics: Qt 6: Blank rectangles (mainly with AMD graphics?) #7398 (seems to be fixed in Qt 6.4)
• Mouse wheel sometimes stops working for me: Qt6: mouse scrolling doesn't work after suspend until qb is restarted #7523 (known upsteam Qt issue)
• favicons don't show for some tabs sometimes (TODO: find reproducer, going to eg https://github.com/qutebrowser/qutebrowser/commits/qt6-test and clicking on one of the commits seems to lose the favicon) (from @toofar, haven't checked if it's still an issue) see Switching to Qt 6 by default #7202 (comment)
• Wrong MPRIS name: Qt 6: Wrong MPRIS name #7431

TODOs

• Get macOS builds to work
• Look at remaining API changes and new features (comment below) moved to Interesting changes in Qt 6.0/.1/.2/.3 #7756
• Look at remaining FIXME:qt6 comments
• Make sure qutebrowser has enough feedback (e.g. logging?) about which Qt wrappers it uses and why
• Maybe investigate storing the Qt wrapper to use in a config file so that we don't unexpectedly switch (and destroy data?) if one is unavailable
• Decide what to do about test coverage (switch to Qt 6? Combine?)
• Probably merge some PRs while waiting for more feedback and to avoid crazy conflicts with the next steps moved to v4
• Integrate Feat/pyqt wrapper bulk import rewrite #7128 if we didn't find a way to make static analysis work otherwise see Type checking and auto-completion with PyQt6 (and PyQt5) #7370 and linked discussion and issue
• Adopt black (Use auto-formatters (black/isort, maybe autopep8/yapf) #1455) so that we pass "normal" linting again and don't have crazy long lines moved to v4
• Look at adopting PySide6 again not a priority at the moment
• Check if we can revert 3c4d51a
• Potentially start cherry-picking some stuff to master
• Figure out why our way of blocking back/forward mouse button presses broke
• Perhaps introduce a Qt 5 warning to get people to switch over

[The-Compiler]

Remaining interesting changes

Moved to #7756

Remaining API changes overview

Moved to #7756

[The-Compiler]

@toofar I just found this:

• [QTBUG-103735] QWebEnginePage::iconChanged() not emitted - Qt Bug Tracker
• Do not reset favicon for same document navigations (Iee0c09cd) · Gerrit Code Review

That seems like a fix for the favicon problem you observed. Not sure if we can add a workaround of some sorts.

[toofar]

Yay, another case of waiting for someone else to fix it paying off! It looks like that fix is being picked to the 6.2 and 6.3 branches, so no need for a workaround?

[donaldguy]

EDIT: Well ... I'm not gonna say this isn't possibly interesting eventually, but I suppose its largely irrelevant right now, as ... something I saw somewhere said 3.8 wasn't good enough for Qt6 anyway(?), and 3.9 out of homebrew is not quite so much of a mess

---

TL;DR: a (if unlikely the) macOS build problem is that, using Apple's Python, the virtualenv construction is creating a .venv/bin/python3 that fails in dylib loading, as the lib is not found successfully relative to the binary's (weird macOS-style) canonical location outside the path

---

Apologies if this is covered already somewhere (and if so I'd suggest some sort of mention should be in docs, if not indeed in the script):

I am taking a run at getting qt6-v2 to build on macOS. Still kinda just started, but insofar as the current latest state I see is

PyInstaller doesn't seem to work properly

I wanted to point out a thing that's like ... not extremely obvious (and very ... Apple)

When running ./scripts/mkvenv.py --pyqt-type source, it SIGABORTs pretty early; a crashdump shows:

Termination Reason:    Namespace DYLD, Code 1 Library missing
Library not loaded: @executable_path/../Python3
Referenced from: /Users/USER/*/python3
Reason: tried: '/Users/donald/qutebrowser/.venv/bin/../Python3' (no such file), '/usr/local/lib/Python3' (no such file), '/usr/lib/Python3' (no such file)
(terminated at launch; ignore backtrace)

This is cause

macOS /usr/bin/python3 (amongst others) is weird (and fake) as:

1. it is inside the sealed, local, read-only APFS / mount
2. Apple still wants it to be able to be upgraded and redirected in parallel with other parts of an SDK/toolchain.

So yeah whereas

❯ /usr/bin/python3 -c "import sys; print(sys.executable)"
/Applications/Xcode.app/Contents/Developer/usr/bin/python3

even this is a lie, cause the reality is that, for the linking to work out, it seems like the canonical location ends up as:

• (in my immediate case) /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8
• (or if I wanted) /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8
• (or more flexibly) $(xcode-select -p)/Library/Frameworks/Python3.framework/Versions/Current/bin/python3 (with Current and python3 as symlinks to a sibling directory, so linking is consistent)

---

This is to say, on macOS python3 (unless pre-empted) is apt to resolve to /usr/bin/python3, which is a wrapper program

❯ otool -L /usr/bin/python3
/usr/bin/python3:
	/usr/lib/libxcselect.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.120.1)

which execs python in such a way that its understood argv0 is the python3 from the usr/bin of the active /Developer directory, which in turn is a symlink (I'd have thought hard but I'd be wrong) to the real location (of the python3 which is itself a symlink to the dotted version) within the .framework

(And honestly, it may actually be supposed to ignore the disk entirely and hit up the in-memory dylib cache? but it still probably needs the canonical path whether or not that corresponds to a real file)

---

For my part I worked around this by doing

(cd $(git rev-parse --show-toplevel);
  rm .venv/bin/python{,3}; 
  ln -s $(xcrun -f python3) .venv/bin/; 
  ln -s $(xcrun -f python3) .venv/bin/python)
) && ./scripts/mkvenv.py --keep

doing this but with hardlinks recreates the issue; so going from scratch you gotta like

python3 -m venv --symlinks .venv
./scripts/mkvenv.py --keep --pyqt-type source

which gets me back to good old fashioned qt errors; adding --qt-version 6.3 is getting me more progress now

[donaldguy]

Well, I don't know if any of that turns out to be extremely relevant or not - but I do have a Qt6 6.3 M1 build running!

Here is the initial version info: https://paste.the-compiler.org/view/9ec20cf5

---

This seems to get me back to a working browser!

# getting to scratch: a thing I surely won't regret!
brew uninstall --ignore-dependencies --force qt || :
brew uninstall --ignore-dependencies --force python
rm -rf /opt/homebrew/lib/python3.* 
rm -rf ~/Library/Caches/pip

brew cleanup
rm -rf ~/Library/Caches/Homebrew/downloads/*

# for bonus points, kill the qt deps with fire:
brew uninstall --force --ignore-dependencies assimp brotli dbus double-conversion freetype hunspell icu4c jasper jpeg libb2 libmng libpng libtiff md4c sqlite webp zstd 
# pcre2 is a dep of git, so it probs can be assumed if brew is working
# maybe also don't wanna kill off glib, just for sanity

brew install python # 3.9.13_1, also brings in sqlite @ 3.38.5
gh repo clone qutebrowser/qutebrowser
cd qutebrowser
git checkout qt6-v2
/opt/homebrew/opt/python@3.9/libexec/bin/python scripts/mkvenv.py --pyqt-version 6.3
./.venv/bin/qutebrowser --temp-basedir

This builds like this https://paste.the-compiler.org/view/884b4209

it did generate arm64 is not a supported architecture for PyQt binaries on darwin, this will most likely fail. but also Successfully installed PyQt6-6.3.0 PyQt6-Qt6-6.3.0 PyQt6-WebEngine-6.3.0 PyQt6-WebEngine-Qt6-6.3.0 PyQt6-sip-13.3.1

Likewise, Smoke test failed with status -11. You might find additional information in the debug output above. (and sometimes(?) a SIGABORT calling back to above? nope looks like a 0-access segfault in QtCore); but also the browser opens and at least basic stuff works

(Probably same-ish version info, but from "clean" build as described: https://paste.the-compiler.org/view/56e3a68d)

[The-Compiler]

@donaldguy

EDIT: Well ... I'm not gonna say this isn't possibly interesting eventually, but I suppose its largely irrelevant right now, as ... something I saw somewhere said 3.8 wasn't good enough for Qt6 anyway(?), and 3.9 out of homebrew is not quite so much of a mess

As in Python 3.8 you mean? That should work fine (anything 3.7 to 3.10 should).

TL;DR: a (if unlikely the) macOS build problem is that, using Apple's Python, the virtualenv construction is creating a .venv/bin/python3 that fails in dylib loading, as the lib is not found successfully relative to the binary's (weird macOS-style) canonical location outside the path

I don't think that's the problem in my case. My python3 (on macOS 10.14.6 Mojave though) points to my brew-installed one, /usr/local/bin/python3. I'm guessing this could be a problem with macOS Monterey (12.3) or newer though, as I think there, Python 2 was removed. Unfortunately, I'm always running the oldest macOS version supported by qutebrowser.

In fact, running python3 scripts/mkvenv.py --pyqt-version 6.3 gives me a working browser too (other than the failing smoke test, which indeed seems like a segfault on exit of some sorts... sigh).

The trouble starts when trying to package it into a dmg: With tox installed, all that should be needed is a tox -e build-release-qt6 -- --skip-docs if you want to reproduce (which would be much appreciated!).

Then, when running open dist/qutebrowser.app or dist/qutebrowser.app/Contents/MacOS/qutebrowser, qutebrowser launches, but the renderer process crashes (you only see a grey page) - with the terminal claiming:

dyld: Library not loaded: @rpath/QtOpenGL.framework/Versions/A/QtOpenGL
  Referenced from: /Users/florian/proj/qutebrowser/dist/qutebrowser.app/Contents/MacOS/PyQt6/Qt6/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess
  Reason: image not found
08:27:19 ERROR: Renderer process crashed (status 6)

and indeed the only thing in the .app is Contents/MacOS/PyQt6/QtOpenGL.abi3.so and Contents/MacOS/QtOpenGL, while the virtualenv install has a .../site-packages/PyQt6/Qt6/lib/QtOpenGLWidgets.framework/Versions/A/QtOpenGLWidgets indeed.

I think it's probably just a matter of convincing PyInstaller to include it...

[donaldguy]

Yeah I was just getting to that bit myself (as you were typing it). Like you said, its kinda just a "someone needs to write the hook" thing... but also it doesn't not seem like the confusion I got into with python 3.8's resolution (vis-a-vis ... weird path dependent linking mismatching argv0's in practice)

---

I do have one new and hopefully useful piece of information for you!

(other than the failing smoke test, which indeed seems like a segfault on exit of some sorts... sigh).

this segfault occurs (only) when QT_DEBUG_PLUGINS is set!

That is:

 ./.venv/bin/python -c 'from qutebrowser.qt.widgets import QApplication; QApplication([])'

exits 0, whereas

QT_DEBUG_PLUGINS=1 ./.venv/bin/python -c 'from qutebrowser.qt.widgets import QApplication; QApplication([])'

consistently segfaults (and so, so does mkvenv.py as written)

I figure there is a chance this is a ~race condition on the draw of the empty app window, but maybe/probably its just the debug output choking on a null string that's found its way into the plugin lookup path? (messing with QT_QPA_PLATFORM_PLUGIN_PATH and QT_PLUGIN_PATH didn't fix it tho)

---

And Yeah, because I had exported QT_DEBUG_PLUGIN=1 at some point, my scripts/dev/build_release.py runs were also consistently segfaulting at smoketest - that's why I took so long to (get any .app that I could look at) and get to seeing the OpenGL error message 😅

I did earlier notice that during the PyInstaller run there is a stream of

121 WARNING: Cannot find path ./QtQuick3DParticleEffects.framework/Versions/A/QtQuick3DParticleEffects (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick3D/ParticleEffects/libqtquick3dparticleeffectsplugin.dylib)
123 WARNING: Cannot find path ./Qt3DQuickScene2D.framework/Versions/A/Qt3DQuickScene2D (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene2D/libqtquickscene2dplugin.dylib)
123 WARNING: Cannot find path ./Qt3DRender.framework/Versions/A/Qt3DRender (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene2D/libqtquickscene2dplugin.dylib)
123 WARNING: Cannot find path ./Qt3DCore.framework/Versions/A/Qt3DCore (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene2D/libqtquickscene2dplugin.dylib)
123 WARNING: Cannot find path ./QtConcurrent.framework/Versions/A/QtConcurrent (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene2D/libqtquickscene2dplugin.dylib)
123 WARNING: Cannot find path ./Qt3DQuick.framework/Versions/A/Qt3DQuick (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene2D/libqtquickscene2dplugin.dylib)
130 WARNING: Cannot find path ./QtQmlLocalStorage.framework/Versions/A/QtQmlLocalStorage (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/LocalStorage/libqmllocalstorageplugin.dylib)
143 WARNING: Cannot find path ./QtQmlXmlListModel.framework/Versions/A/QtQmlXmlListModel (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQml/XmlListModel/libqmlxmllistmodelplugin.dylib)
156 WARNING: Cannot find path ./Qt3DAnimation.framework/Versions/A/Qt3DAnimation (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene3D/libqtquickscene3dplugin.dylib)
156 WARNING: Cannot find path ./Qt3DLogic.framework/Versions/A/Qt3DLogic (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene3D/libqtquickscene3dplugin.dylib)
156 WARNING: Cannot find path ./Qt3DInput.framework/Versions/A/Qt3DInput (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene3D/libqtquickscene3dplugin.dylib)
156 WARNING: Cannot find path ./Qt3DRender.framework/Versions/A/Qt3DRender (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene3D/libqtquickscene3dplugin.dylib)
156 WARNING: Cannot find path ./Qt3DCore.framework/Versions/A/Qt3DCore (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene3D/libqtquickscene3dplugin.dylib)
156 WARNING: Cannot find path ./QtConcurrent.framework/Versions/A/QtConcurrent (needed by /Users/donald/qutebrowser/.tox/pyinstaller-64-qt6/lib/python3.9/site-packages/PyQt6/Qt6/qml/QtQuick/Scene3D/libqtquickscene3dplugin.dylib)

Most (though definitely not all) of these are to do with the 3D libs, which I note have separate PyQt6-3D (and PyQt63D-Qt6) packages (but no corresponding PyInstaller hook in the upstream). I tried adding them to the requirements to little effect

Def could be a red herring or so, but since there is, apparently, direct linking from PyQt6 (core) to them (and given how they are generally related) - it doesn't seem impossible to me that there is some need of them to resolve this path

(Also more broadly it seems like the .frameworks should end up in qutebrowser.app/Contents/Frameworks... but oh well)

I also wondered if there is a hidden import amongst the already available hooks that we want in PyQt6.QtOpenGLWidgets?

and like whether any of the Qml stuff needs to come along to satisfy the plugin resolution?

---

I don't really know Qt at all, and python is also not my forte so this is all a lot; hopefully I am being more helpful than annoying

but anyway, I realized the sun is rising here as 04:30 just passed so, uh, I should go to bed now.

Cheers