Supporting Qt 6

[The-Compiler]

Continued in #7202, most of the information below is purely historical.

---

The next Qt release after 5.15 will be Qt 6. The initial schedule plans for a release 2020-12-01, with feature freeze on 2020-08-31.

QtWebEngine is coming back with Qt 6.2. Release schedule:

• Feature freeze 2021-06-07
• Alpha 2021-06-23
• Beta 1 2021-07-6
• Beta 2 2021-07-27
• Beta 3 2021-08-17
• Beta 4 2021-08-31
• RC 2021-09-14
• Final 2021-09-30

Links:

• Interesting JIRA issues
• Interesting codereview changes: dev, 6.2

API changes

Probably going to be a bunch, but rather small changes... The plan for qutebrowser is to stay compatible with Qt 5 for a while, while gaining compatibility with Qt 6.

• Rename QWebEngineDownloadItem -> QWebEngineDownloadRequest
• Remove obsolete, deprecated api from qwebenginedownloaditem
• Unify certificate error handling
• Move QMenu handling from page to view
• Merge all webenginecontextmenudata wrappers into one class
• Improve QWebEngineCertificateError api
• Remove default qwebenginesettings getter
• Move qwebenginesettings to core
• Move QWebEngineFullScreenRequest to core
• Move to qwebenginescript to core
• Move qtwebenginescriptcollection class to core
• Cleanup qwebenginescript and qwebenginescriptcollection api
• Drop dependency on default notification presenter in profile
• Drop dependency on page in profile (internal only?)
• Move profile from widgets to core
• Remove deprecated url interoceptor setter from profile
• Move client certificate selection to core
• Remove qwebengineview setter from qwebenginepage
• Move QWebEngineHistory to core
• Move QWebEnginePage to core
• Add Qt WebEngine version getter
• Make handling of certificate errors possible from a signal
• Add missing FileSelectionMode enum value
• Move createWindow to a signal (internal only?)
• Move QQuickWebEngineNavigationRequest to Core
• Merge WebEngineHistory's quick implementation into QWebEngineHistory
• QWebEnginePage: add new 'loading' property and expose QWebEngineLoadingInfo
• Move printing finished callback to signal
• Add API for favicon database
• Remove left-over deprecated API QWebEngineProfile::setRequestInterceptor()
• Add QPageRanges to PDF printing
• Make default profile off the record
• Remove deprecated useforglobalcertificateverification

API review follow ups

• Remove duplicated 'downloadProgress' in favor of separate signals (Ib825d63c) · Gerrit Code Review
• Update parameters to findText callback (I622ff55c) · Gerrit Code Review
• Add QWebEngineNewWindowRequest::openIn(QWebEnginePage *) (I15e72d52) · Gerrit Code Review
• Add QWebEnginePage::view() replacement (I552f0047) · Gerrit Code Review
• Return printToPdf to QWebEnginePage (I0e14563c) · Gerrit Code Review

Interesting non-API changes

• Fix inconsistent number of load signals and their order (not an API change, but relevant for us)
• Implement dialogs of File System Access API (no API change, just potentially interesting)
• Add basic USB device logging (same)
• Suppress error pages also for http errors if they are disabled (no API change, but interesting)
• Disable kAllowContentInitiatedDataUrlNavigations (I9398f54b) · Gerrit Code Review
• Widgets (like the command text box) now insert test into the primary selection (if available) (ignore the fix version, it only made it into 6.0 but didn't get a new ticket)

Planned changes in PyQt

• PyQt5 -> PyQt6 rename (see Add wrapper modules for PyQt #995)
• Enums won't be accessible unscoped, only scoped - scoped access is possible since Qt 5.11 I believe, so if we want to access enum members without some unwieldy enum_value(QWebEnginePage, 'FindFlag', 'FindBackward') wrapper, we'll need to drop older versions at that point (Dropping support for Qt < 5.12 (affects Debian Stretch/Buster, Ubuntu 18.04) #3839) - split off to Rewrite enum access for scoped PyQt enums #5904.

More from the website:

PyQt6 introduces the following incompatibilities over PyQt5:

• The Qt module has been removed.
• QtWidgets.qApp has been removed.
• The QtCore.PYQT_CONFIGURATION dict has been removed.
• pyrcc has been removed.
• Scoped enums are now implemented as enum.Enum (PyQt5 used enum.IntEnum). Named enums are now implemented as enum.IntEnum (PyQt5 used a custom type). PyQt5 allowed an int whenever an enum was expected. PyQt6 requires the correct type (ie. it will not accept an int instead of an enum).
• All exec_() methods have been removed.
• All print_() methods have been removed.
• Python v3.6 or later is required.

Changes which probably didn't make it

• Add QWebEngineurlRequestBody (Ia0e4b5a5) · Gerrit Code Review
• Add a new signal "launchExternalProtocol" when launching external url (I744b6ca9) · Gerrit Code Review
• Clear the cache in time range (If80bba20) · Gerrit Code Review
• Add QWebEngineurlRequestBody (Ia0e4b5a5) · Gerrit Code Review
• Add QWebEnginePage::httpStatusCode() (I350c003e) · Gerrit Code Review

[The-Compiler]

Apparently the Qt 6 release is going to be "essential" Qt modules only, i.e. not QtWebEngine. That will then be part of Qt 6.1. Instead, the underlying Chromium will be upgraded for Qt 5.15 patch releases, though it's unclear whether those will be commercial-only at the point Qt 6.0 is released...

If Qt 6.1 is following the usual schedule, that'd be around May/June 2021. Probably doesn't make much sense for qutebrowser to start migrating things long before that. On the positive side, that buys us some time for all the other things that should be done for a v2.0.0 release.

[The-Compiler]

FWIW the initial QtWebEngine API review is out now: https://codereview.qt-project.org/c/qt/qtwebengine/+/312867

[The-Compiler]

A clarification on the timeline for QtWebEngine:

Note QtWebengine also meant to be independent of Qt releases in the same time frame, so might not make that much sense to promise it with Qt releases. We hope to have a QWE release ready around the same time as 6.1, but the official goal is with 6.2.

Judging from the 5.14 release, that'd mean between May/June 2021 (6.1) and December 2021/January 2022 (6.2). Still plenty of time until then... Still no details (as far as I can tell) on what the independent release means for e.g. PyQt (and thus qutebrowser) exactly.

[The-Compiler]

From a Qt Development mailinglist mail today:

We have long discussed, that the timing of our feature releases to be just before summer and Christmas vacation is a bit unfortunate, as we have little slack for delays without going into the vacation period. Especially the releases in December have sometimes been difficult in that respect. So we’d like to push the schedule a bit and move the minor releases towards a Spring/Autumn schedule.

A somewhat shorter release cycle directly after 6.0 is probably a good idea anyway, as we will probably still need to do changes/fixes that don’t quite fit with our policy for patch level releases.

So the idea is to shorten the release cycle for Qt 6.1 a bit and focus mainly on bug fixing and stability for that release. We’d aim for a feature freeze by the end of January, and a final Qt 6.1.0 release end of April.

6.2 would then also happen a bit earlier, with a feature freeze in June and a release end of September.

Content wise, I believe we’ll start seeing more and more of the add-ons from Qt 5 being supported over the next 6-9 months, and I believe that with Qt 6.2 we will have brought most modules that we supported in Qt 5.15 over to Qt 6.

[The-Compiler]

There's finally some more official communication around this now: https://www.qt.io/blog/add-on-support-in-qt-6.0-and-beyond

[The-Compiler]

Nothing left here for v2.0.0 - I've done some trivial changes; the enums change (#5904) needs bigger changes to the mypy stubs first and won't be in v2.0.0.

[The-Compiler]

QtWebEngine API review: https://codereview.qt-project.org/c/qt/qtwebengine/+/356324

Notable changes I can see:

• New qWebEngineVersion() and qWebEngineChromiumVersion() (6.3 also adds qWebEngineChromiumSecurityPatchVersion())
• Some QWebEngineCertificateError changes, but IIRC we don't use that yet
• QWebEngineDownloadItem renamed to QWebEngineDownloadRequest
  • DownloadType / type() removed, but we don't use that
  • Deprecated path() and setPath() removed
  • More signals we could potentially use?
  • finished signal renamed to isFinishedChanged
• QWebEngineFullScreenRequest now uses some kind of callback?
• QWebEngineHistory changed, now has a QWebEngineHistoryModel?
• QWebEnginePage changes
  • QWebEnginePage::FileSelectionMode::FileSelectUploadFolder
  • QWebEnginePage::view() and setView() removed
  • Most callbacks in QWebEnginePage changed to std::function, but that hopefully doesn't affect us via Python
  • Print API was moved to page?
  • New certificateError signal, emitting QWebEngineCertificateError (replaces certificateError virtual)
  • New navigationRequested signal, emitting QWebEngineNavigationRequest (new class)
  • New newWindowRequested signal, emitting QWebEngineNewWindowRequest (new class)
  • New loadingChanged signal, emitting QWebEngineLoadingInfo (new class)
• QWebEngineProfile changes
  • Deprecated QWebEngineProfile::setRequestInterceptor removed (qutebrowser should already be adjusted)
  • New notificationPresenter() getter
• QWebEngineScript changes
  • isNull() removed
  • New sourceUrl() and setSourceUrl()
  • QWebEngineScriptCollections.size() removed (duplicates .count())
  • QWebEngineScriptCollections.findScript() and .findScripts() replaced by .find()
• QWebEngineSettings changes
  • globalSettings() moved (?)
  • defaultSettings() removed
  • New setParentSettings

[The-Compiler]

Well, looks like a pyqt6-webengine appeared, and I did some hacking:

It barely works though, I spent an hour or so just to get it to the point where it wouldn't crash outright. There are some issues with PyQt6 I noticed, so it's full of hacks, and probably slightly insecure too (URL passwords will get logged/exposed where they weren't before).

Definitely requires a lot more work to be fully usable, and of course it will also need to stay compatible with Qt 5...

Either way unfortunately a lot of changes are required to make things work with the new enum scoping and imports. Even for a quick and dirty hack, it's around 2000 changed lines. Thus, lots of PRs will conflict... I plan to take a look at some low-hanging fruit before starting to work on Qt 6 support in master, but at the same time, I don't want to hold back things once Qt 6.2 is out, as staying on an older Chromium with Qt 5.15 leads to compatibility and security issues.

[The-Compiler]

There's some more progress in the qt6-test branch thanks to @toofar - known issues:

• Various fun around enum handling
• Missing qutebrowser icon resources (Use importlib.resources for icons #6062)
• The issue you reported about UrlFormattingOption and ComponentFormattingOption isn't entirely resolved, you can combine them now (with |) but you can't pass ComponentFormattingOption values to the methods that claim to accept a UrlFormattingOption but actually should accept both.
• Category headings in the completion dialog don't look fancy anymore, selected completion row's background is blue
• qenum_key and friends probably can be simplified since we're now dealing with real Python enums.
• QWebEngineHistory serialization doesn't work anymore, for instance when closing a tab. A non fatal error from Qt is printed out QVariant::save: unable to save type 'QObject*' (type id: 39). When you try to undo the tab close then the deserialization will choke on the incomplete data stream. The individual history items can be serialized fine though. Saving and loading sessions still seems to work fine. I'm not sure whether this is a Qt or PyQt issue, I thought it might be https://codereview.qt-project.org/c/qt/qtwebengine/+/327625 (parent of QWebEngineHistory is now a QObject) but I tried to reproduce in C++ and didn't get the same error (it also didn't print out as much hex as I was expecting though, so I might have messed something else up).
• Lots more in my original changes, I'm sure

[The-Compiler]

New features in Qt 6+

• [QTBUG-93848] QNetworkInformation: Behind connectivity portal - Qt Bug Tracker

[The-Compiler]

I see someone™ has reported the serialization issue to PyQt and it was resolved - relevant changelog entry:

2021-11-22  Phil Thompson  <phil@riverbankcomputing.com>

	* NEWS, PyQt6.msp:
	Added /Constrained/ to the QVariant serialiser so that it doesn't
	act as a catchall and bypass all subsequent serialisers (including
	slot extenders).
	[ce7d2be67940] <6.2-maint>

Makes me wonder if there is some kind of workaround we could do, but guess not...

I've also worked a bit more on the qt6-test branch today and noticed some more issues:

• PyInstaller builds seem to fail with RuntimeError: PyQt6.QtWebEngineWidgets cannot import type '\ufffda\ufffd:\ufffd�' from PyQt6.QtCore. No clue what that means...
• qutebrowser's JS seems to not run or something? e.g. using f says "Unknown error while getting elements" and the JS console reveals Cannot read property 'webelem' of undefined, which sounds like window._qutebrowser isn't defined. Possibly a bug or some change in QWebEngineScript? @toofar, did you look into that one so far?
• We use .path() for downloads which is now removed ("# FIXME use 5.14 API")

[The-Compiler]

Note to self:

<saro> The-Compiler is it expected c.qt.force_software_rendering = 'qt-quick' to crash qt6-v2 ?

edit: [QTBUG-103372] [REG 6.2 -> 6.3] QT_QUICK_BACKEND=software segfaults with QtWebEngine in QSGSoftwareRenderableNode::update - Qt Bug Tracker and 9a6ce83 - cc @saro

[The-Compiler]

@tinywrkb I just pushed another fix, could you give it a spin? Fingers crossed that it will work this time around....

---

Also, quick status update: qt6-v2 is coming along nicely, and from an user's perspective, should be fine to test now really. I think I fixed the last known crashes (the last one was when I was trying to print a website, guess nobody tried that so far 🤣). Next up is a rebase of the >200 commits in the branch to end up with a nice overview of what was actually changed why, and then we "only" need to figure out how to make it fit all together (#7128, some other major open PRs I'd really like to merge, reformatting everything, etc.)...

[tinywrkb]

No problem!
KDE 6.3 runtime should be available soon, so we will have to run the tests again with PyQt6.

I took the liberty to publish beta builds with PyQt6 through the flathub-beta repo.
It's QtWebEngine 6.2 of course, so it will be a bit more interesting when the KDE 6.3 runtime will be out.

The beta branch uses the same app-id as the stable channel (master branch), so the warning about data dir applies here.
I don't see a way to change datadir. If there's one, then I can add a wrapper to set it to something like $XDG_DATA_HOME/qutebrowser-beta, if it's even something that we want.

No automated "nightly" rebuilds yet, this is being updated manually. I'll try writing a GitHub workflow for this.

Adding the flathub-beta remote and installing from the beta branch can be done through an application manager (Gnome Software, Discover, ...) or via the command-line.

$ flatpak remote-add --if-not-exists flathub-beta https://flathub.org/beta-repo/flathub-beta.flatpakrepo
$ flatpak install flathub-beta org.qutebrowser.qutebrowser//beta

[The-Compiler]

I don't see a way to change datadir. If there's one, then I can add a wrapper to set it to something like $XDG_DATA_HOME/qutebrowser-beta, if it's even something that we want.

You could either use --basedir (though that also affects config/cache, which might not be what you want here), or just override XDG_DATA_HOME.

[tinywrkb]

Overriding XDG_DATA_HOME is also not something I want to do. I'll just leave it as it is.
It would have been nice to have something like the CHROME_VERSION_EXTRA variable to append a - + version to qutebrowser's config and data dir names.

[The-Compiler]

Another quick status update:

• qt6-v2 has been rebased on the latest master now. There were quite some conflicts, so please let me know if you notice any regressions.
• I also managed to adjust build scripts to produce windows builds: qutebrowser-2.5.0-amd64-qt6.exe, qutebrowser-2.5.0-windows-standalone-amd64-qt6.zip
• No macOS builds yet. PyInstaller doesn't seem to work with Qt 6 on macOS yet. Will need to investigate...

---

@tinywrkb See #4102 re some ideas about having multiple named data profiles. Probably an argument (and/or envvar) would be a good idea to have, once that exists.

[The-Compiler]

This has a lot of important historic information which I'd rather not hide, but it's still a lot of stuff in this issue now. Thus, let's continue in #7202 where I tried to summarize everything that seemed still relevant.