Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crashed on search in log files with 5 million lines #200

Open
yurivict opened this issue Dec 24, 2017 · 4 comments
Open

Crashed on search in log files with 5 million lines #200

yurivict opened this issue Dec 24, 2017 · 4 comments

Comments

@yurivict
Copy link

The search progresses to 40%, then slows down and crashes.
FreeBSD 11.1

@variar
Copy link
Contributor

variar commented Dec 25, 2017

@yurivict could you upload your file somewhere and provide search pattern that caused that crash? Might be some FreeBSD issue.

@yurivict
Copy link
Author

yurivict commented Dec 25, 2017

I selected 5 sample lines from the log file. Please run this shell script:

echo "[16:05:30] Site:1Gfey7..fcdp Try to get updates from: Peer:59.78.171.105 Left: [<Peer:74.82.60.33 >, <Peer:36.149.3.27>, <Peer:110.185.40.144>, <Peer:82.132.224.203>, <Peer:217.20.210.113>, <Peer:74.82.160.65 >, <Peer:207.229.128.38>, <Peer:101.229.178.5>, <Peer:171.15.127.81>, <Peer:220.118.79.25>, <Peer:154.16.127.99>, <Peer:131.21.181.186>, <Peer:119.4.121.73>, <Peer:221.125.176.145>, <Peer:219.147.140.204>, <Peer:74.82.60.164 >, <Peer:113.57.33.71>, <Peer:36.51.179.142>, <Peer:221.130.50.206>, <Peer:212.169.111.134>, <Peer:123.139.86.70>, <Peer:201.185.39.103>, <Peer:131.219.222.194>]" > fake.log
echo "[16:05:30] FileServer Conn# 5 174.182.60.33  [?] > Connecting..." >> fake.log
echo "[16:05:44] Site:1CpQfw..GP2o Announced types ['onion'] in mode startup to 4 trackers in 10.001s, errors: [], slow: ['1.74s http://tracker.opentrackr.org:1337/announce', '1.36s http://explodie.org:6969/announce', '10s+ http://tracker1.wasabii.com.tw:6969/announce']:" >> fake.log
echo "[16:05:46] Site:138R53..NXLP Http tracker tracker1.wasabii.com.tw:6969/announce error: 0x06: TTL expired" >> fake.log
echo "[16:05:55] Site:1BLogC..AGg8 183.14.133.178:15441 Getting connection error: SOCKS5Error: 0x06: TTL expired in Peer.py line 83 > ConnectionServer.py line 152 (connection_error: 1, hash_failed: 0)" >> fake.log

(for i in `seq 100000`; do cat fake.log fake.log fake.log fake.log fake.log fake.log fake.log fake.log fake.log fake.log; done) > fake5M.log

Then search for a common pattern there: TTL.
It stars very fast, but then slows down progressively, and never finishes.

@variar
Copy link
Contributor

variar commented Dec 25, 2017

As far as I understand the slowdown is mostly due to matches overview (the area near vertical scrollbar that shows approximate matching line positions). It updates on UI thread and processes all matches on every update thus getting slower during search. Refactoring this behavior is on my todo list for next klogg milestone (this is my fork, see issue 18). I'll do PR for glogg if code changes are not very extreme.

Unfortunately I can't reproduce the crash with either glogg or klogg on my linux box. Did you build glogg from source? Was it 32- or 64-bit version? Could you reproduce the crash under debugger to get call stack?

@yurivict
Copy link
Author

I couldn't reproduce crashes when I tried today either. Unfortunately, I deleted the original file. But this slowdown is pretty bad as it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants