Freeze when search in history (:open)

[AckslD]

Version info:
qutebrowser v3.0.0
Git commit:
Backend: QtWebEngine 6.5.2, based on Chromium 108.0.5359.220 (from api)
Qt: 6.5.2

CPython: 3.11.3
PyQt: 6.5.2

Qt wrapper info:
PyQt6: success
PyQt5: not imported
-> selected: PyQt6 (via autoselect)

colorama: no
jinja2: 3.1.2
pygments: no
yaml: 6.0.1
adblock: no
objc: no
PyQt6.QtWebEngineCore: 6.5.0
PyQt6.sip: 6.7.10
pdf.js: 3.7.107 (/usr/share/pdf.js/build/pdf.js)
sqlite: 3.43.0
QtNetwork SSL: OpenSSL 3.1.2 1 Aug 2023

Style: QFusionStyle
Platform plugin: xcb
OpenGL: Intel, 4.6 (Compatibility Profile) Mesa 23.1.6
Platform: Linux-6.4.12-arch1-1-x86_64-with-glibc2.38, 64bit
Linux distribution: Arch Linux (arch)
Frozen: False
Imported from /usr/lib/python3.11/site-packages/qutebrowser
Using Python from /usr/bin/python3
Qt library executable path: /usr/lib/qt6, data path: /usr/share/qt6

Paths:
cache: /home/axel/.cache/qutebrowser
config: /home/axel/.config/qutebrowser
data: /home/axel/.local/share/qutebrowser
runtime: /run/user/1000/qutebrowser
system data: /usr/share/qutebrowser

Autoconfig loaded: no
Config.py: /home/axel/.config/qutebrowser/config.py has been loaded
Uptime: 0:00:24

Does the bug happen if you start with --temp-basedir?:
No (since there is no history)

Description
I'm not sure if this is a duplicate of eg #4099 or not but when I search for certain patterns when opening site/tab with eg :open or :open -t, qutebrowser completely freezes for sometimes minutes and is unresponsive. I'm not really sure how the history loading works but it's not on the initial load but rather when I type in some specific pattern. It's also very deterministic. Is this a known issue?

I'm not fully sure how to provide enough input on this without sharing my history-file. What I tried though is to see if there is a lot of entries for that pattern. For example, if I type :open dock I get the freeze, but not for eg :open doc nor :open docr. But if I try to check how many such entries there actually are in the history with

strings history.sqlite | rg dock

I only get 244 matches. The total size of the history file is 29 MB.

How to reproduce
Happy to get an ideas of what I can try to reproduce this without sharing my private history file. Also let me know if there already is a similar issue tracked somewhere. I think even if the searching is slow it would be ideal if it at least was not blocking and uninteruptable.

[toofar]

[completion related] qutebrowser completely freezes for sometimes minutes and is unresponsive

That sounds painful. We did have a report of the completion performing inconstantly in #7862 which was somehow linked to using the "xos4 Terminus" bitmap font. Could you check what font you are using and if changing it (via fonts.default_family) to something like "DejaVu Sans Mono" and seeing if that performs better>

history querying

I would be surprised if that had regressed, but you never know! Long strings have caused various issues before, also emoji I guess. You could get some idea of where long string might be like:

sqlite3 history.sqlite
.mode csv
.headers on
with match as (select '%dock%' as pat) select count(*) as matches, avg(url_length), max(url_length), avg(title_length), max(title_length) from (select length(url) as url_length, length(title) as title_length from CompletionHistory, match where url like match.pat or title like match.pat);

Not sure about emojis, probably something like sqlite3 history.sqlite "select title from CompletionHistory where title like '%dock%';" | grep -P -n "[\x80-\xFF]". Although maybe a higher range, I'm seeing lots of "middle dot"
on my output.

more testing

To see if it is disk IO (although with a 30M DB file I doubt it) I like to use netdata because it can show system time in IO wait and utilization per disk and you can zoom and enhance it. You could also look at fatrace I suppose.

If you could run py-spy record on it while you reproduce this issue that could be informative. It's presumably doing something when it is frozen.

Getting debug logs could show something too, maybe.

[AckslD]

Thanks a lot @toofar, that's very helpful! Btw, I also found #3989 which this also might be a duplicate of.

Changing the font does not seem to change anything.

However it seems like I indeed have a very long entry since from your command I got:

matches,avg(url_length),max(url_length),avg(title_length),max(title_length)
36,96.7777777777778,391,191.777777777778,4096

I also think it might be the same one with a lot (!) of unicode characters.

This is maybe better suited for a discussion of forum but if you don't mind me asking, how can I go about removing this entry?

[AckslD]

Okay this is definitely the culprit! I think at some point I was looking into using a python package which in turn depended on https://pypi.org/project/china-dictatorship/. It seems that their github page has a very long title with a lot of unicodes:
https://github.com/cirosantilli/china-dictatorship

Now I just need to figure out how to remove it from the history.

[AckslD]

Adding that single url to completion.web_history.exclude works 🎉

[toofar]

Now I just need to figure out how to remove it from the history.

FWIW you can use Ctrl+D when the completion item entry is highlighted to remove it from history.

It's good that it's got faster for you now. I'm annoyed I can't reproduce though. I might get some people with similar reports to try to use your reproducer too.

[AckslD]

Thanks for the tip about Ctrl+D @toofar!