Switching to Qt 6 by default

[The-Compiler]

(Continuation of #5395 because that was getting rather big)

Current state

On July 22th, 2023, the flip was finally switched to Qt 6 by default. All you need to ensure is what PyQt6 is available, and Qt 6 will be used automatically. If it's not available, you will get a warning shown by qutebrowser.

All information below is considered historical.

The qt6-v2 branch has been merged to master on March 18th, 2023, but with Qt 5 still being the default. See #7628 for details. This issue is now repurposed on what there needs to be done for Qt 6 to be the default Qt wrapper.

The last fixes for the qt6-v2 branch with Qt 6 support went in around May 19th. Multiple core developers (and users) have been using it as a daily driver since far longer (late 2021 / early 2022) and all known crashes are fixed, with all tests passing.

There is still a lot of remaining work, however, on the development side. Due to the big amount of changes (currently 156 commits, 314 files changed, 12,978 additions and 4,094 deletions) it's tricky to integrate this into master without breaking lots of existing open PRs. Thus, we'd like to do another round of PR merging before attempting to do so. The current approach also breaks various tooling, by not properly supporting static analysis (#7128, but that will introduce another big code churn), or by not fixing linting issues yet (#1455).

Bottomline: Feel free to use the qt6-v2 branch, keep in mind it will be rebased from time to time, and it will still take lots of efforts behind the scenes to get it fully integrated.

How to try

mkvenv.py

• Take a backup of your qutebrowser data directory (see :version) unless you don't mind losing cookies and such when down-/upgrading. This isn't needed if you use --temp-basedir below, but if you forget to do so, your data will get upgraded, and as far as I'm aware, there is no way to read it with Qt 5 after that.
• Clone the git repo, e.g. via git clone https://github.com/qutebrowser/qutebrowser.git
• cd qutebrowser
• Check out the branch using git checkout qt6-v2
• Create a new separate environment via python3 scripts/mkvenv.py --venv-dir .venv-qt6 --pyqt-version 6.3
• Set QUTE_QT_WRAPPER=PyQt6 somewhere in the environment
• Run .venv-qt6/bin/qutebrowser --temp-basedir and verify via :version that you're indeed on Qt 6

Archlinux

qutebrowser-qt6-git is available in the AUR.

Windows / macOS

Built as part of the nightly builds on CI. Click on the most recent passing build, then find the qt6 builds under "Artifacts". Note that you will need to be logged in with a GitHub account to download.

Note for macOS: Due to a PyInstaller issue, Chromium's sandboxing is currently disabled for macOS builds with Qt 6. This means that there will be no additional layer of protection in case of Chromium security bugs. Thus, it's advised to not use this build in production. Hopefully, this situation will be resolved before the final 3.0.0 release.

Flatpak

Published in the flathub-beta repo:

Adding the flathub-beta remote and installing from the beta branch can be done through an application manager (Gnome Software, Discover, ...) or via the command-line.

$ flatpak remote-add --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo
$ flatpak install flathub-beta org.qutebrowser.qutebrowser//beta

Remaining issues

• Flickering rectangles with AMD graphics: Qt 6: Blank rectangles (mainly with AMD graphics?) #7398 (seems to be fixed in Qt 6.4)
• Mouse wheel sometimes stops working for me: Qt6: mouse scrolling doesn't work after suspend until qb is restarted #7523 (known upsteam Qt issue)
• favicons don't show for some tabs sometimes (TODO: find reproducer, going to eg https://github.com/qutebrowser/qutebrowser/commits/qt6-test and clicking on one of the commits seems to lose the favicon) (from @toofar, haven't checked if it's still an issue) see Switching to Qt 6 by default #7202 (comment)
• Wrong MPRIS name: Qt 6: Wrong MPRIS name #7431

TODOs

• Get macOS builds to work
• Look at remaining API changes and new features (comment below) moved to Interesting changes in Qt 6.0/.1/.2/.3 #7756
• Look at remaining FIXME:qt6 comments
• Make sure qutebrowser has enough feedback (e.g. logging?) about which Qt wrappers it uses and why
• Maybe investigate storing the Qt wrapper to use in a config file so that we don't unexpectedly switch (and destroy data?) if one is unavailable
• Decide what to do about test coverage (switch to Qt 6? Combine?)
• Probably merge some PRs while waiting for more feedback and to avoid crazy conflicts with the next steps moved to v4
• Integrate Feat/pyqt wrapper bulk import rewrite #7128 if we didn't find a way to make static analysis work otherwise see Type checking and auto-completion with PyQt6 (and PyQt5) #7370 and linked discussion and issue
• Adopt black (Use auto-formatters (black/isort, maybe autopep8/yapf) #1455) so that we pass "normal" linting again and don't have crazy long lines moved to v4
• Look at adopting PySide6 again not a priority at the moment
• Check if we can revert 3c4d51a
• Potentially start cherry-picking some stuff to master
• Figure out why our way of blocking back/forward mouse button presses broke
• Perhaps introduce a Qt 5 warning to get people to switch over

[donaldguy]

EDIT: Well ... I'm not gonna say this isn't possibly interesting eventually, but I suppose its largely irrelevant right now, as ... something I saw somewhere said 3.8 wasn't good enough for Qt6 anyway(?), and 3.9 out of homebrew is not quite so much of a mess

---

TL;DR: a (if unlikely the) macOS build problem is that, using Apple's Python, the virtualenv construction is creating a .venv/bin/python3 that fails in dylib loading, as the lib is not found successfully relative to the binary's (weird macOS-style) canonical location outside the path

---

Apologies if this is covered already somewhere (and if so I'd suggest some sort of mention should be in docs, if not indeed in the script):

I am taking a run at getting qt6-v2 to build on macOS. Still kinda just started, but insofar as the current latest state I see is

PyInstaller doesn't seem to work properly

I wanted to point out a thing that's like ... not extremely obvious (and very ... Apple)

When running ./scripts/mkvenv.py --pyqt-type source, it SIGABORTs pretty early; a crashdump shows:

Termination Reason:    Namespace DYLD, Code 1 Library missing
Library not loaded: @executable_path/../Python3
Referenced from: /Users/USER/*/python3
Reason: tried: '/Users/donald/qutebrowser/.venv/bin/../Python3' (no such file), '/usr/local/lib/Python3' (no such file), '/usr/lib/Python3' (no such file)
(terminated at launch; ignore backtrace)

This is cause

macOS /usr/bin/python3 (amongst others) is weird (and fake) as:

1. it is inside the sealed, local, read-only APFS / mount
2. Apple still wants it to be able to be upgraded and redirected in parallel with other parts of an SDK/toolchain.

So yeah whereas

❯ /usr/bin/python3 -c "import sys; print(sys.executable)"
/Applications/Xcode.app/Contents/Developer/usr/bin/python3

even this is a lie, cause the reality is that, for the linking to work out, it seems like the canonical location ends up as:

• (in my immediate case) /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8
• (or if I wanted) /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8
• (or more flexibly) $(xcode-select -p)/Library/Frameworks/Python3.framework/Versions/Current/bin/python3 (with Current and python3 as symlinks to a sibling directory, so linking is consistent)

---

This is to say, on macOS python3 (unless pre-empted) is apt to resolve to /usr/bin/python3, which is a wrapper program

❯ otool -L /usr/bin/python3
/usr/bin/python3:
	/usr/lib/libxcselect.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.120.1)

which execs python in such a way that its understood argv0 is the python3 from the usr/bin of the active /Developer directory, which in turn is a symlink (I'd have thought hard but I'd be wrong) to the real location (of the python3 which is itself a symlink to the dotted version) within the .framework

(And honestly, it may actually be supposed to ignore the disk entirely and hit up the in-memory dylib cache? but it still probably needs the canonical path whether or not that corresponds to a real file)

---

For my part I worked around this by doing

(cd $(git rev-parse --show-toplevel);
  rm .venv/bin/python{,3}; 
  ln -s $(xcrun -f python3) .venv/bin/; 
  ln -s $(xcrun -f python3) .venv/bin/python)
) && ./scripts/mkvenv.py --keep

doing this but with hardlinks recreates the issue; so going from scratch you gotta like

python3 -m venv --symlinks .venv
./scripts/mkvenv.py --keep --pyqt-type source

which gets me back to good old fashioned qt errors; adding --qt-version 6.3 is getting me more progress now

[donaldguy]

Well, I don't know if any of that turns out to be extremely relevant or not - but I do have a Qt6 6.3 M1 build running!

Here is the initial version info: https://paste.the-compiler.org/view/9ec20cf5

---

This seems to get me back to a working browser!

# getting to scratch: a thing I surely won't regret!
brew uninstall --ignore-dependencies --force qt || :
brew uninstall --ignore-dependencies --force python
rm -rf /opt/homebrew/lib/python3.* 
rm -rf ~/Library/Caches/pip

brew cleanup
rm -rf ~/Library/Caches/Homebrew/downloads/*

# for bonus points, kill the qt deps with fire:
brew uninstall --force --ignore-dependencies assimp brotli dbus double-conversion freetype hunspell icu4c jasper jpeg libb2 libmng libpng libtiff md4c sqlite webp zstd 
# pcre2 is a dep of git, so it probs can be assumed if brew is working
# maybe also don't wanna kill off glib, just for sanity

brew install python # 3.9.13_1, also brings in sqlite @ 3.38.5
gh repo clone qutebrowser/qutebrowser
cd qutebrowser
git checkout qt6-v2
/opt/homebrew/opt/python@3.9/libexec/bin/python scripts/mkvenv.py --pyqt-version 6.3
./.venv/bin/qutebrowser --temp-basedir

This builds like this https://paste.the-compiler.org/view/884b4209

it did generate arm64 is not a supported architecture for PyQt binaries on darwin, this will most likely fail. but also Successfully installed PyQt6-6.3.0 PyQt6-Qt6-6.3.0 PyQt6-WebEngine-6.3.0 PyQt6-WebEngine-Qt6-6.3.0 PyQt6-sip-13.3.1

Likewise, Smoke test failed with status -11. You might find additional information in the debug output above. (and sometimes(?) a SIGABORT calling back to above? nope looks like a 0-access segfault in QtCore); but also the browser opens and at least basic stuff works

(Probably same-ish version info, but from "clean" build as described: https://paste.the-compiler.org/view/56e3a68d)

[The-Compiler]

I don't think the missing Qt3D libraries are an issue, as we don't use those parts of QML/QtQuick. In an effort to keep this issue a bit more streamlined than the previous ones, I've split the QT_DEBUG_PLUGINS segfault into a separate issue:

• Qt 6: Segfaults on macOS with QT_DEBUG_PLUGINS #7257

As for the PyInstaller issues, I believe I got to the root cause:

• Fix QtWebEngine with PyQt6 framework builds pyinstaller/pyinstaller#6892

I seem to get a working DMG that way, with some warnings remaining:

• Qt 6: Figure out OpenSSL situation on macOS #7255
• Qt 6: Resource warnings with PyInstaller on macOS #7256

Still need to clean up things a bit, and then I'll upload it for people to try out.

[The-Compiler]

FYI, did another rebase on master yesterday - should be all fine now after some (short-lived) trouble I had. A few new tests are failing (#7267), but I don't think that's an actual bug, seems more likely to be an issue with the tests of some sorts.

[The-Compiler]

macOS and Windows builds of the qt6-v2 branch are now available via the nightly builds on CI. See the original post for more information, especially when you're on macOS.

Planning to release v2.5.2 now, and then do another rebase on the latest master.

[tinywrkb]

I tried today QtWebEngine v6.4.0-beta1 with Chromium 98.0.4758.90 built against the KDE 6.3 Flatpak runtime (Qt 6.3.x) and WebRTC screen sharing with PipeWire seems to be broken.
The log does show that Qt added the WebRTCPipeWireCapturer, but nothing else from I can tell from a quick glance, and not a single relevant message went through the D-Bus session.
Maybe will have to backport some patches. I think I remember there was something around Chromium 99 or 100 related to PipeWire, but I might remember it wrong. I don't remember if it was this or something else.

edit: This is possibly a QtWebEngine bug, as I suspect that Qt doesn't pass flags to Chromium

edit2: Found it!

[tinywrkb]

The beta channel of the Flathub app is now (finally) updated to QtWebEngine 6.3.1 with Chromium 94.0.4606.126+.

[tinywrkb]

Flatpak test builds with development releases of QtWebEngine 6.4 (Chromium 98.0.4758.90) are available here.

I believe that this is the first time building QtWebEngine with the app, and not pulling in the QtWebEngine Flatpak base application.
Testing building QtWebEngine with the app was on my to-do list, so I used the 6.4-beta1 release as an excuse.
This has the benefits of having control on the QtWebEngine version, the Chromium revision, and being able to update QtWebEngine quicker.
The drawbacks are more complicated and much longer build, and compiling this locally is not an option for me ATM.

The test builds are temporary CI artifacts, and they will need to be generated every week or so.
The current test build is missing the sandboxing patches that make it possible to sandbox render processes, but the next test build will likely to have them.

The warning about data directory still applies here when switching between 6.3 and 6.4.

[angelus2014]

Hello,

I'm not sure if I should report this here or make a seperate issue. I tried qutebrowser-nightly-2022-07-24-dfde979-qt6-macos yesterday and it seems to run fine without any issues. But I did notice however that the memory usage is a lot bigger than with the current stable version. I'm talking about 3-4 times as much memory is being used for the same pages open.

I'm running it on a MB Air (M1), OS 12.5.

---

With regards,

Angelo Machils

[rnhmjoj]

Websites are starting to become unusable: I think it's time to switch, even if the Qt 6 branch is not 100% finished.

[The-Compiler]

I'm aware (and the branch doesn't really exist anymore, support is already in master). Unfortunately Qt 6.5.1 had some rather bad regressions, so I'm waiting for 6.5.2 (scheduled for the 28th).

I want to avoid a situation where we flip the switch and people instantly want to go back due to regressions. For one because they'll lose their cookies and such when doing that, but also because they might end up configuring Qt 5 and then never switching back to Qt 6 even if those things are long fixed.

[The-Compiler]

Next big step done: #7741 is now merged, which in a nutshell means:

• Useful feedback about what Qt wrapper was chosen and why, everywhere it matters (e.g. :version and error dialogs)
• New --qt-wrapper argument to qutebrowser which can be used to switch
• New QUTE_QT_WRAPPER=auto which enables auto-detection (PyQt6 by default, PyQt5 if that is not available)

Final step will be making that the default, planning to do that when Qt 6.5.2 did land. Maybe I'll find the time to check some of the remaining Qt 6 regressions (personally annoyed by #7222 most) in the meantime...

[The-Compiler]

Well, I'm ready for when Qt 6.5.2 is out!

• Flip the Qt 6 switch #7755

[The-Compiler]

FYI, I deleted the old qt6-v2 and qt6-test branches to avoid people using old stuff without realizing it's stale.

They're preserved at qutebrowser/experiments if we ever need them in the future.

[The-Compiler]

Still waiting for Qt 6.5.2 to be released. It's been planned for Wednesday, but it looks like things are delayed. I found this:

• Update submodule refs on '6.5.2' in yocto/meta-qt6 (If7af82fa) · Gerrit Code Review
• Update submodules on '6.5.2 in qt/qt5' (I3d71880c) · Gerrit Code Review

Soooo maybe it'll still happen today? Or on Monday?

[The-Compiler]

Well, looks like the domain that's hosted on just did a derp:

As of January 1, 2014 the Internet Corporation for Assigned Names and Numbers (ICANN) will mandate that all ICANN accredited registrars begin verifying the Registrant WHOIS contact information for all new domain registrations and Registrant contact modifications.

Why this domain has been suspended
Email address has not been verified.
This is a new domain registration and the Registrant email address has not been verified.

or
The Registrant contact data for this domain was modified but still requires verification.
Specifically the First Name, Last Name and/or email address have been changed and never verified.

Soooo I'm assuming it'll probably be next week 😅

[DemiMarie]

I strongly recommend dropping Qt5 support in the very near future for security reasons, as Qt5 WebEngine is behind on patches.

[The-Compiler]

I strongly recommend dropping Qt5 support in the very near future for security reasons, as Qt5 WebEngine is behind on patches.

Can you elaborate? The last security patches for QtWebEngine on Qt 5 were integrated four days ago.

[DemiMarie]

Was not aware of that, sorry.

[The-Compiler]

As for Qt 6.5.2, it looks like there was another issue found during releasing:

• [QTBUG-115150] shadow build failure, webengine - Qt Bug Tracker
• [QTBUG-115067] [REG 6.5.1 -> 6.5.2] shadow build failure, webengine - Qt Bug Tracker
• Speculative fix for potential race condition in devtools build (Icf342eb2) · Gerrit Code Review

Welp. Let's hope it still happens this week, as next week I'm at Europython and probably not going to have the time to look at qutebrowser stuff much. Fingers crossed!

[The-Compiler]

Finally there's been some news:

unfortunately, Qt 6.5.2 has been delayed. The final test round for the release candidate is currently ongoing, though. My current estimate is that we can release 6.5.2 at the beginning of next week if there are no critical findings on the remaining tests. I’ll update this also to the Qt’s wiki page – I’m sorry that I didn’t do it already earlier (Jani Heikkinen who’s usually managing Qt 6.5.x releases is currently on summer holiday and I’m substituting for him).

And the wiki page now says 18th (next Tuesday).

[The-Compiler]

Qt 6.5.2 has been released finally! Looks like an Archlinux update also is in progress already, so I suppose it'll happen somewhen the next days (perhaps on the weekend at the Europython sprints?) ✨

[The-Compiler]

Since Qt 6.5.2 was released finally (with 3 weeks of delay), fixing various annoying issues with Qt 6.5.0 and .1, it's time to finally flip the switch to Qt 6 by default!

So that's what I did at the Europython sprints in Prague 🙂

Right now mkvenv.py will install the PyQt6 Qt6 packages from the Riverbank Computing development package server (which means you end up downloading it twice). This should change early next week, when those packages get officialy released, but I just couldn't wait any longer 😆

Thanks to @RMPR for some nice pair programming sessions and contributions here at the sprint, fixing some of the last rough edges.

If you use qutebrowser from git, it will now automatically use PyQt6 if that's available - if it is not, it will open a warning page telling you that you should probably ensure that it is.

Only a couple of smaller things now remaining until the v3.0.0 release: https://github.com/qutebrowser/qutebrowser/milestone/49

[The-Compiler]

soooooooon! Hopefully today!

[The-Compiler]

v3.0.0 was released!

[markstos]

Perhaps this can be unpinned now that Qute 3 / QT 6 are the default.

[The-Compiler]

Leaving it pinned for now, until/unless we need space for new pinned issues. While I don't see many people still running Qt 5, I still think it's good to make people aware that there was a big change and they should make sure they're on Qt 6 if they haven't taken care of that yet.