Browser freezes when attempting to download

[amosbird]

Version info:

qutebrowser v2.5.0
Git commit: 2cbefa1 on qt6-v2 (2022-05-30 22:00:13 +0800)
Backend: QtWebEngine 6.3, based on Chromium 94.0.4606.126
Qt: 6.3.0

Does the bug happen if you start with --temp-basedir?:
Yes

Description

It's very slow to do downloading on the qt6 branch. I have to wait for at least 20 seconds to see the download prompt after clicking some download button, and during
the time, CPU reaches 100%

How to reproduce

Try downloading any resource (perhaps >= 1GB) .

[rien333]

Using qtwebengine 6.3, I can't reproduce a freeze, but 'downloading' versus 'not downloading' a 2.66GB file with ~5Mb/s causes a 40-75% CPU usage increase in the python3 /usr/bin/qutebrowser process (as measured by htop). Initially, I did indeed reach ~100% CPU, but after a while, it fell down to on average of about 50%.

This does seem a bit high. However, it doesn't at all freeze the browser, nor does it slow down my laptop in general.

I have to wait for at least 20 seconds to see the download prompt after clicking some download button

Not sure if I follow; maybe you could include a concrete example/URL?

[toofar]

I also couldn't reproduce slow download starts, downloading linux ISOs (qt6-v2, QtWebEngine 6.3, based on Chromium 94.0.4606.126, X11).

(I got somewhat distracted by the CPU usage while a download was going on, #5376 helps a little bit because there are lots of QTimerEvent(0) triggers for the GUI thread, but I don't think the CPU usage is in our code in the end, maybe)

[rien333]

I did indeed reach ~100% CPU, but after a while, it fell down to on average of about 50%.

Not seeing this anymore. Linux ISOs start at >40%, and then stays there with occasional drops into 20%. By comparison, wget uses about 9% CPU on my machine. Using qtwebengine 6.3.1.

[The-Compiler]

FWIW I'm sometimes seeing this: There is a few seconds of delay before the download dialog opens. I saw it with Qt 5 as well, but only when there are hundreds/thousands of files in the download folder. It seems to happen more often (but not always) with Qt 6 to me, even with a nearly empty downloads folder.

[rien333]

Ah right, I have no download dialog, but just always download to ~/Downloads. Maybe that's why I don't really see a freeze. My bad for not reading OP correctly.

[The-Compiler]

Python stacktrace when this happens:

File "/home/florian/proj/qutebrowser/git/qutebrowser/utils/qtutils.py", line 478 in exec
  File "/home/florian/proj/qutebrowser/git/qutebrowser/mainwindow/prompt.py", line 200 in ask_question
  File "/home/florian/proj/qutebrowser/git/qutebrowser/utils/message.py", line 273 in ask
  File "/home/florian/proj/qutebrowser/git/qutebrowser/browser/webengine/webenginedownloads.py", line 323 in handle_download
  File "/home/florian/proj/qutebrowser/git/qutebrowser/app.py", line 140 in qt_mainloop
  File "/home/florian/proj/qutebrowser/git/qutebrowser/app.py", line 

C++ stack seems to be:

#0  __GI___access (file=0x55d42ff1e420 "/home/florian/.local/share/icons/hicolor/24x24@2/stock/net/folder.svg", type=0) at ../sysdeps/unix/sysv/linux/access.c:27
#1  0x00007fe0ac060147 in QFileSystemEngine::fillMetaData(QFileSystemEntry const&, QFileSystemMetaData&, QFlags<QFileSystemMetaData::MetaDataFlag>) (entry=..., data=..., what=...)
    at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/corelib/io/qfilesystemengine_unix.cpp:1012
#2  0x00007fe0abedc8b0 in QFileInfo::exists(QString const&) (file=<optimized out>) at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/corelib/global/qflags.h:110
#3  0x00007fe0a7f2b86e in QIconLoader::findIconHelper(QString const&, QString const&, QList<QString>&) const
   Python Exception <class 'gdb.error'>: cannot resolve overloaded method `end': no arguments supplied
 (this=0x7fe0a85c8180 <QGlobalStatic<QtGlobalStatic::Holder<(anonymous namespace)::Q_QGS_iconLoaderInstance> >::instance()::holder>, themeName=<optimized out>, iconName="", visited=...)
    at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/gui/image/qiconloader.cpp:492
#4  0x00007fe0a7f3497e in QIconLoader::loadIcon(QString const&) const (this=0x7fe0a85c8180 <QGlobalStatic<QtGlobalStatic::Holder<(anonymous namespace)::Q_QGS_iconLoaderInstance> >::instance()::holder>, name="")
    at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/corelib/tools/qarraydata.h:88
#5  0x00007fe0a7f34ae5 in QIconLoaderEngine::ensureLoaded() (this=this@entry=0x55d430417bf0) at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/gui/image/qiconloader.cpp:627
#6  0x00007fe0a7f34cbd in QIconLoaderEngine::ensureLoaded() (this=0x55d430417bf0) at /usr/src/debug/build/include/QtGui/6.3.1/QtGui/private/../../../../../../qtbase-everywhere-src-6.3.1/src/gui/image/qiconloader_p.h:183
#7  QIconLoaderEngine::isNull() (this=0x55d430417bf0) at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/gui/image/qiconloader.cpp:829
#8  0x00007fe0a7f24650 in QIcon::fromTheme(QString const&) (name="") at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/gui/image/qicon.cpp:1325
#9  0x00007fe0a7f1c30b in QAbstractFileIconProviderPrivate::getIconThemeIcon(QAbstractFileIconProvider::IconType) const (this=this@entry=0x55d42b75a970, type=type@entry=QAbstractFileIconProvider::Folder)
    at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/gui/image/qabstractfileiconprovider.cpp:132
#10 0x00007fe0a7f1c42e in QAbstractFileIconProviderPrivate::getIconThemeIcon(QFileInfo const&) const (this=0x55d42b75a970, info=...) at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/gui/image/qabstractfileiconprovider.cpp:155
#11 0x00007fe0a8330c18 in QAbstractFileIconProvider::icon(QFileInfo const&) const (info=..., this=<optimized out>) at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/gui/image/qabstractfileiconprovider.cpp:262
#12 QFileInfoGatherer::getInfo(QFileInfo const&) const (this=0x55d41e19f3e8, fileInfo=...) at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/gui/itemmodels/qfileinfogatherer.cpp:332
#13 0x00007fe0a8343e97 in QFileSystemModelPrivate::_q_fileSystemChanged(QString const&, QList<std::pair<QString, QFileInfo> > const&) (this=<optimized out>, path=<optimized out>, updates=<optimized out>)
    at /usr/src/debug/qtbase-everywhere-src-6.3.1/src/gui/itemmodels/qfilesystemmodel.cpp:1935

with the file paths pointing to /home/florian/tmp, which might have been my last remembered download directory there, and does indeed contain some 9000 files.

The C++ stack seems to hang there pretty consistently judging by a quick watch -n0.1 "eu-stack -p 5062 -mid | head -n 20".

@amosbird could it be something similar for you, with your download directory containing a lot of files?

[amosbird]

with your download directory containing a lot of files?

Nope, only several files.

[The-Compiler]

Weirdly enough, for me, it looks like all Qt applications on my system freeze while I'm waiting for the download dialog to open?! e.g. Telegram Desktop also doesn't respond anymore for a while after clicking a qutebrowser download link.

[toofar]

Seems like this happens when choosing a file to upload too, when using the default Qt dialog and navigating it to a folder with lots of files. Strace is showing logs of groups of files accesses like:

statx(143, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0600, stx_size=1557, ...}) = 0
access("/home/user/.local/share/mime/text/plain.xml", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/user/.local/share/mime/text/plain.xml", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
access("/home/user/.local/share/flatpak/exports/share/mime/text/plain.xml", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/user/.local/share/flatpak/exports/share/mime/text/plain.xml", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/tmp/tmp.SeijStZrLn", {st_mode=S_IFREG|0600, st_size=14941, ...}) = 0
openat(AT_FDCWD, "/tmp/tmp.SeijStZrLn", O_RDONLY|O_CLOEXEC)

Seems like it could be the same as this issue: https://bugreports.qt.io/browse/QTBUG-110416

[fstecker]

I am having the same problem recently: whenever I click on a download link, the entire qutebrowser (including e.g. a video playing in another tab) freezes for a few seconds before the download dialog is displayed. My ~/Downloads folder contains >3000 files, and indeed this freezing does not happen if I replace the ~/Downloads folder with an empty one.

Looking at strace, I see hundreds of thousands of accesses to nonexistent files like
/home/user/.local/share/icons/hicolor/32x32@2/emotes/application-x-java-archive.png
It seems like Qutebrowser/Qt iterates through every file in the ~/Downloads folder and tries to find an icon for each file type, by trying to access hundreds of possible file names for the icon. Which seems like a pretty strange method to find file icons even for a "normal" file dialog, but the one in Qutebrowser doesn't even display any icons! I don't know Qt well, is there maybe a "use no Icons" Option of some sorts? Or do you think we need to get this fixed in Qt?

[fstecker]

After poking around in the source a bit, I get the impression that the use of QFileSystemModel in qutebrowser/mainwindow/prompt.py is the culprit. That loads a lot more information about all files in the folder (maybe also icons?), but Qutebrowser chooses to display only the filename.

https://doc.qt.io/qt-6/qfilesystemmodel.html#caching-and-performance

This claims that QFileSystemModel runs the lookup in another thread and that it caches the information. Either of those things should prevent this behaviour, but it doesn't seem to be happening?

[The-Compiler]

I agree it's probably something with the QFileSystemModel, yep. Thanks for the digging!

I wonder if the QFileSystemModel::DontUseCustomDirectoryIcons option would change anything. It's probably files rather than directories which are the problem for us, yet it says:

Always use the default directory icon. Some platforms allow the user to set a different icon. Custom icon lookup causes a big performance impact over network or removable drives.

[fstecker]

I tried the QFileSystemModel.Options.DontUseCustomDirectoryIcons option, it didn't seem to make a difference. Opening the file dialog produced around 260,000 syscalls in total with or without the option.

As an alternative, I tried setting the icon provider via QFileSystemModel.setIconProvider to a dummy class that returns the standard text file icon for any file type. That reduced the number of syscalls to between 7000 and 8000. A couple hundred of these are for looking up the text file icon, and then there are about 7000 of these:

newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=3552, ...}, 0) = 0

No idea what that is supposed to do. But at least for my ~3000 files in ~/Downloads this works well enough and makes the delay unnoticeable.

[The-Compiler]

@fstecker Thanks for taking a look! I think this warrants some more detailed investigation before going for a band-aid fix, but any chance you could post a diff here or perhaps open a PR?

[fstecker]

Overloading the icon method of QAbstractFileIconProvider got rid of the bulk of syscalls (reduced it from ~260,000 to ~13,000). The remaining ones were related to mime type lookup, so I also overloaded the type method, making it return an empty string. While that did remove the mime lookups, it somehow caused Qt to access /etc/localtime twice for every file. I'm still not sure why it does that. But it turns out that if type returns some nonempty string like 'unknown', these syscalls also go away. That's what the current version of the PR does, and with that all per-file syscalls are gone.

The only alternative I can think of right now would be to not use QFileSystemModel at all, and to query the directory entries some other way.

[The-Compiler]

Looks like this is pretty much known upstream, but nobody seems to care too much:

• [QTBUG-99178] QFileSystemModel should have an option to disable icon loading - Qt Bug Tracker
• [QTBUG-78779] call to QFileDialog::getOpenFileName really slow on WSL Ubuntu 19.04 - Qt Bug Tracker
• [QTBUG-27806] QFileSystemModel hang fetching file icons in main thread - Qt Bug Tracker

So yeah, I suppose this is an appropriate solution, short of writing completely custom file system handling.

[toofar]

I'm going to mark this as fixed by #7925 for now.

I'm confident that we identified and resolved one issue that was affected several users (you've got lots of files in your download directory, the download dialog is slow to respond). It's not 100% clear if everyone who contributed to this issue was running into the same thing or if there is some other issue. We can re-open this or a new issue as appropriate in that case.
Note that there is still this issue that occured on Wayland in that past (not sure if it's still an issue on Qt6): #4857