Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[US$ 1500 bounty] Big TX history causing crash & freezes #6625

Open
kennedyiowa opened this issue Oct 1, 2020 · 27 comments
Open

[US$ 1500 bounty] Big TX history causing crash & freezes #6625

kennedyiowa opened this issue Oct 1, 2020 · 27 comments
Labels
performance 🏎 topic-wallet 👛 related to wallet.py, or maybe address_synchronizer.py/coinchooser.py user-interface 🔲

Comments

@kennedyiowa
Copy link

kennedyiowa commented Oct 1, 2020 •
edited
Loading

With big wallets containing a lot of transactions, Electrum will become unbearably slow, lag, freeze and eventually crash. #2540 is possibly related. My wallet size is 50MB by now, but it started way earlier, probably 10MB.

Version 3.3.8 is the last Electrum version where I can still open my wallet. Starting with Electrum 4 (I tried 4.0.3), Electrum won't even open anymore, the process is running but nothing happens. The experience below is from 3.3.8 (but earlier versions were affected, too):

Doing some work with the API, def update_wallet(self): is running very very slow as well.

Electrum runs fine for a minute, freezes completely for 10-20 seconds, then works fine again. After running Electrum for about 1 day in the background, eventually it will synchronize endlessly (it finishes, then starts to synchronize again immediately), resulting in a crash after doing that for a certain amount of time.

If I had to take a guess, the lags and freezes probably result from Electrum loading and showing all transactions at once. It is possible that having these loaded and drawn as needed could fix the issue.

Electrum-LTC has the same issue.

The only active command I have:
wallet.change_gap_limit(1000)

My PC is fast and uses an NVME M.2 SSD.

Since this kind of load would only burden public servers, I'm using my own Electrumx server and node.

While these are bugs and if sufficiently documented, you could probably fix it; the bounty is primarily here because its an edge case and so I don't have to bother with handing you over a wallet that reproduces this. Maybe you can generate a big wallet via Testnet or import a lot of addresses.

The bounty will be valid for 2 months from now. The goal is to fix the freezes and that Electrum 4 cannot even open the wallet. If you think you can tackle this issue and the 2 months are over or need more time, just reply and I can renew it, depending on the situation.

If you are unsure whether this bounty is legitimate or not, just reply below posting status updates to this issue and additionally shortly before its ready to be PR'd. I can make the transaction before your work is delivered. US$ 1500 in XMR or BTC.
@kennedyiowa kennedyiowa changed the title [US$ 1500 bounty] Fix Electrum Crash & Lag resulting from TX history [US$ 1500 bounty] Fix Electrum Crash & Lag resulting from big TX history Oct 1, 2020
@kennedyiowa kennedyiowa changed the title [US$ 1500 bounty] Fix Electrum Crash & Lag resulting from big TX history [US$ 1500 bounty] Big TX history causing crash & freezes Oct 1, 2020
@SomberNight SomberNight added performance 🏎 topic-wallet 👛 related to wallet.py, or maybe address_synchronizer.py/coinchooser.py user-interface 🔲 labels Oct 5, 2020
@SomberNight
Copy link
Member

How many addresses and transactions are there in your wallet?
In the Console tab, run these (the results are just examples):

>>> len(wallet.db.transactions)
104
>>> len(wallet.get_addresses())
145

@kennedyiowa
Copy link
Author

Electrum BTC:

>> len(wallet.db.transactions)
10000
>> len(wallet.get_addresses())
85000

Electrum LTC:

>> len(wallet.db.transactions)
2000
>> len(wallet.get_addresses())
45000

The BTC one is a few months old wallet. The issues started after a few days or a week already, so the values would've been much lower back then.

@nochiel
Copy link

nochiel commented Nov 10, 2020

@kennedyiowa: A few questions:

Version 3.3.8 is the last Electrum version where I can still open my wallet. Starting with Electrum 4 (I tried 4.0.3),

Is electrum attempting to update an old wallet? What's the output of get('seed_version')
If electrum hangs while trying to upgrade the wallet, then I can write a converter in C.

I haven't been able to synthesise a large enough testnet wallet to reproduce your bug, however, based on a similar experience with an in-house asset editor that made extensive use of json, I am attempting the following (which may be dumb but, without relevant detailed profiling data, it's the only experiment I can think of):

  • Get rid of all use of json for the wallet. Instead of storing wallet as compressed, encrypted text, store wallet as a binary blob corresponding to in-memory runtime structs of arrays.
  • On startup, convert json wallet to binary blob and write it to disk.

@SomberNight
Copy link
Member

Don't try to blindly fix it; it would be a waste of time. You need to be able to reproduce first.

Electrum runs fine for a minute, freezes completely for 10-20 seconds, then works fine again

I can confirm this. It's easily reproducible with very large wallets.

Starting with Electrum 4 (I tried 4.0.3), Electrum won't even open anymore, the process is running but nothing happens

Not sure what changed between 3.3.8 and 4.0.x that could cause this.
I can still open my very large wallets fine.
@kennedyiowa how are you running Electrum? From source or from official binaries? If binaries, which one? Which OS?

@Transisto
Copy link

Transisto commented Nov 10, 2020 via email

@nochiel
Copy link

nochiel commented Nov 10, 2020 •
edited
Loading

Electrum runs fine for a minute, freezes completely for 10-20 seconds, then works fine again

I can confirm this. It's easily reproducible with very large wallets.

@SomberNight Does the daemon log indicate when the freeze occurs? Is #6697 the fix for that?

@SomberNight
Copy link
Member

The largest wallet I can publicly share is one I have created, for testnet.

Here is the xpub: vpub5VfkVzoT7qgd5gUKjxgGE2oMJU4zKSktusfLx2NaQCTfSeeSY3S723qXKUZZaJzaF6YaF8nwQgbMTWx54Ugkf4NZvSxdzicENHoLJh96EKg

>>> len(wallet.db.transactions)
11012
>>> len(wallet.get_addresses())
10536

Note that multiple things can contribute to freezes at very large wallet size. Many ~trivial operations become resource hogs after a certain threshold. For example, it helps to disable all fiat-related options.

Does the daemon log indicate when the freeze occurs? Is #6697 the fix for that?

That's certainly somewhat related to large wallets, although it has nothing to do with freezes. The freezes are mainly to do with GUI refreshes.

@kennedyiowa
Copy link
Author

kennedyiowa commented Nov 14, 2020 •
edited
Loading

Apologies for the late reply, I haven't checked this issue daily recently.

@SomberNight Windows 10, downloaded from the official website. Using the installer, since I think this one is faster. To my knowledge, portable is affected, too.

And yes, definitely, the fiat-related options cause a lot of lag as well.

@nochiel Naturally I can only run that command on 3.3.8, the seed version is 18.

In any case, I could just recreate the wallet from seed if the conversion is the issue (correct me if I'm mistaken), but that issue is secondary. In any case, the freezes is the most important one :P

Also, consider this bounty renewed until end of 2020 for now. Of course, like before, if you think you can tackle this issue and the bounty is expired or need more time, just reply and I can renew it, depending on the situation.

@SomberNight
Copy link
Member

If you are using the official Windows binaries with such a large wallet, you might be running out of memory.
The win binaries we build are 32 bit (x86), they can allocate upto ~4 gigs of memory. (see #6598)
The reason 3.3.8 works but 4.0.x does not could be simply due to 4.0.x needing comparatively more memory.

Could you check (using e.g. Task Manager) how much RAM gets allocated by the Electrum process when using 3.3.8?

@kennedyiowa
Copy link
Author

kennedyiowa commented Nov 14, 2020 •
edited
Loading

Could you check (using e.g. Task Manager) how much RAM gets allocated by the Electrum process when using 3.3.8?

3.3.8 is sitting at 1.5gigs, so we are fine there at least.

ln2max pushed a commit to ln2max/electrum that referenced this issue Nov 17, 2020
always use MOST_RELAXED if oneserver is set
8cbbff7 
if oneserver is set then the user is making a deliberate choice to use a
(potentially less performant and/or privately run) server. as we see in
issue spesmilo#6625 these are often cases where the user has a large wallet and
therefore may need a longer timeout for processing requests. The user
should not have to care about whether auto_connect is set in order to
get a longer timeout
@ln2max ln2max mentioned this issue Nov 17, 2020
Closed
@ln2max
Copy link

ln2max commented Nov 17, 2020 •
edited
Loading

@kennedyiowa please give my PR above a spin, it may fix at least this issue:

it will synchronize endlessly (it finishes, then starts to synchronize again immediately), resulting in a crash after doing that for a certain amount of time.

@kennedyiowa
Copy link
Author

kennedyiowa commented Nov 17, 2020 •
edited
Loading

@kennedyiowa please give my PR above a spin, it may fix at least this issue:

Thanks a lot for contributing. Sadly the wallet does not even load on Electrum 4.0 for me and I don't really have the time to backport it to 3 and self compile. For other reasons, including security reasons, I also have to wait until this is merged and released officially.

I will certainly honor fixing this side issue to a small degree when I can confirm this was the issue. Alternatively, you could try loading SomberNight's testnet wallet, leave it running for a few days straight (usually took around 1-3 days for me, now with my bigger wallet its within 12 hours), try to get it to crash and then verify your fix is working :)

@ln2max
Copy link

ln2max commented Nov 19, 2020 via email

@Transisto
Copy link

Transisto commented Nov 19, 2020 via email

@ln2max
Copy link

ln2max commented Nov 19, 2020 via email

@ln2max
Copy link

ln2max commented Nov 19, 2020 via email

@ln2max
Copy link

ln2max commented Nov 21, 2020 •
edited
Loading

Ok, it looks like after many many hours of running, electrum stops refreshing balances at between 50 and 80 addresses into the 10000 or so of the full @SomberNight wallet.

I've got the full verbose console output logged to the attached verbose.log file, the last sync attempt ends at about line 24884 in the file.

I don't see anything in the log which would indicate what the problem is.

Overall the user experience is quite alright, even after the sync stops, Electrum is still responsive and restarting it fixes the issue.

@kennedyiowa since the UX with SomberNight's testnet wallet is fine, it may be worth exporting the xpub (for testing) from your wallet and importing it into Electrum 4 that way -- rather than the wallet itself, Electrum may be choking on importing your wallet. You may find that the only real bug is having to restart the wallet every day, which is an annoyance but not a showstopper.

@kennedyiowa
Copy link
Author

kennedyiowa commented Nov 22, 2020 •
edited
Loading

@kennedyiowa since the UX with SomberNight's testnet wallet is fine, it may be worth exporting the xpub (for testing) from your wallet and importing it into Electrum 4 that way -- rather than the wallet itself, Electrum may be choking on importing your wallet. You may find that the only real bug is having to restart the wallet every day, which is an annoyance but not a showstopper.

Good idea. Just did that on 4.0.5, and apparently the issue wasn't the wallet. 4.x doesn't open at all even in a clean state. Wiped the Electrum appdata folder, started Electrum and it just hangs in the background forever like I mentioned in the original issue. Even reinstalled Windows since I opened this issue. Pretty weird since I don't see any other issues like this on here.

Restarting Electrum every day one or two times is only a small annoyance compared to the freezes. I work with the wallet a lot and need to check transactions. Usually within 60 seconds it will become fully unresponsive for about 10-20 seconds. Then works for another 60 seconds until it repeats. This is costing me more brain cells than the bounty is worth.

@ln2max This freeze issue is not occuring at all on your system? Are you using Windows? Maybe version 4 did fix the freezes, but I wouldn't assume so, otherwise SomberNight wouldn't have confirmed it. Or your changes fixed it, but I really don't see that this was the cause of the freezes.

@ln2max
Copy link

ln2max commented Nov 22, 2020 via email

@SomberNight
Copy link
Member

Starting with Electrum 4 (I tried 4.0.3), Electrum won't even open anymore, the process is running but nothing happens.
Just did that on 4.0.5, and apparently the issue wasn't the wallet. 4.x doesn't open at all even in a clean state

Uhmm, that's weird.

So you are on Windows... could you please try using the setup/installer exe?
It should unpack two executables in the install folder: one should be named e.g. electrum-4.0.5-debug.exe.
Try starting that, from cmd/powershell, with -v, so e.g.:

PS C:\Program Files (x86)\Electrum> .\electrum-4.0.5-debug.exe -v

Anything interesting in the logs there?

@sureplus1988

This comment has been minimized.

Sign in to view
@kennedyiowa
Copy link
Author

Anything interesting in the logs there?

Late again. Its just stuck here (wiped AppData folder, it recreated it):

I | logging | Electrum version: 4.0.5 - https://electrum.org - https://github.com/spesmilo/electrum
I | logging | Python version: 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 18:01:55) [MSC v.1900 32 bit (Intel)]. On platform: Windows-10-10.0.19041-SP0
I | logging | Logging to file: None
I | logging | Log filters: verbosity '*', verbosity_shortcuts ''
I/p | plugin.Plugins | registering hardware bitbox02: ('hardware', 'bitbox02', 'BitBox02')
I/p | plugin.Plugins | registering hardware coldcard: ('hardware', 'coldcard', 'Coldcard Wallet')
I/p | plugin.Plugins | registering hardware digitalbitbox: ('hardware', 'digitalbitbox', 'Digital Bitbox wallet')
I/p | plugin.Plugins | registering hardware keepkey: ('hardware', 'keepkey', 'KeepKey wallet')
I/p | plugin.Plugins | registering hardware ledger: ('hardware', 'ledger', 'Ledger wallet')
I/p | plugin.Plugins | registering hardware safe_t: ('hardware', 'safe_t', 'Safe-T mini wallet')
I/p | plugin.Plugins | registering hardware trezor: ('hardware', 'trezor', 'Trezor wallet')
I/p | plugin.Plugins | registering wallet type ('2fa', 'trustedcoin')
D | util.profiler | Plugins.__init__ 0.0579
I/n | network | blockchains [0]
I | exchange_rate.FxThread | using exchange CoinGecko
I/n | network | starting network
I/n | network | setting proxy None
I/n | network | connecting to blockstream.info:700:s as new interface
I/n | network | starting taskgroup.
I | channel_db.ChannelDB | SQL thread started
D | util.profiler | Daemon.__init__ 1.1162
I | daemon.Daemon | launching GUI: qt
I | channel_db.ChannelDB | Creating database
I | daemon.Daemon | starting taskgroup.
I | gui.qt.history_list | could not import electrum.plot. This feature needs matplotlib to be installed.
I | channel_db.ChannelDB | load data 0 0 0
I | channel_db.ChannelDB | num_channels_partitioned_by_policy_count. 0p: 0, 1p: 0, 2p: 0
D | util.profiler | ChannelDB.load_data 0.0020

@SomberNight
Copy link
Member

SomberNight commented Dec 7, 2020 •
edited
Loading

@kennedyiowa looks like it does not even get to the very early steps of initialising the GUI...

electrum/electrum/daemon.py

Line 569 in 78513af

self.logger.info(f'launching GUI: {gui_name}')

electrum/electrum/gui/qt/__init__.py

Lines 89 to 95 in 78513af

class ElectrumGui(Logger):
@profiler
def __init__(self, config: 'SimpleConfig', daemon: 'Daemon', plugins: 'Plugins'):
set_language(config.get('language', get_default_language()))
Logger.__init__(self)
self.logger.info(f"Qt GUI starting up... Qt={QtCore.QT_VERSION_STR}, PyQt={QtCore.PYQT_VERSION_STR}")

but that's not all... the event loop thread (e.g. network) is not making progress either.

@SomberNight
Copy link
Member

Could you please try running with --offline?
so e.g.:

PS C:\Program Files (x86)\Electrum> .\electrum-4.0.5-debug.exe -v --offline

@kennedyiowa
Copy link
Author

It changes to this:

I | logging | Electrum version: 4.0.5 - https://electrum.org - https://github.com/spesmilo/electrum
I | logging | Python version: 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 18:01:55) [MSC v.1900 32 bit (Intel)]. On platform: Windows-10-10.0.19041-SP0
I | logging | Logging to file: None
I | logging | Log filters: verbosity '*', verbosity_shortcuts ''
I/p | plugin.Plugins | registering hardware bitbox02: ('hardware', 'bitbox02', 'BitBox02')
I/p | plugin.Plugins | registering hardware coldcard: ('hardware', 'coldcard', 'Coldcard Wallet')
I/p | plugin.Plugins | registering hardware digitalbitbox: ('hardware', 'digitalbitbox', 'Digital Bitbox wallet')
I/p | plugin.Plugins | registering hardware keepkey: ('hardware', 'keepkey', 'KeepKey wallet')
I/p | plugin.Plugins | registering hardware ledger: ('hardware', 'ledger', 'Ledger wallet')
I/p | plugin.Plugins | registering hardware safe_t: ('hardware', 'safe_t', 'Safe-T mini wallet')
I/p | plugin.Plugins | registering hardware trezor: ('hardware', 'trezor', 'Trezor wallet')
I/p | plugin.Plugins | registering wallet type ('2fa', 'trustedcoin')
D | util.profiler | Plugins.__init__ 0.0589
I | exchange_rate.FxThread | using exchange CoinGecko
D | util.profiler | Daemon.__init__ 0.0040
I | daemon.Daemon | launching GUI: qt
I | daemon.Daemon | starting taskgroup.
I | gui.qt.history_list | could not import electrum.plot. This feature needs matplotlib to be installed.

@pooler pooler mentioned this issue Mar 26, 2021
Open
@amuna1225
Copy link

Hi all, i'm having this issue on windows 10 with electrum ltc, have had the issue through numerous updates and am currently on 4.0.9.3 and the issue is still present. What an earlier poster said regarding how it freezes for 10-20 seconds then comes back to life is accurate..but eventually it doesnt come back to life at all and i'm forced to restart the software..i have tried using every version (portable, installer, exe, ect) and every single one freezes. My wallet has about 6000 transactions in it.
I have noticed and haven't seen this mentioned that 90% of the time this happens is when a transaction is being sent/received or the server is trying to update my wallet to the next block. I am seriously considering using another wallet for LTC at this point with how often this freezes for me and how annoying and time consuming it is. I run Dash electrum also and have never had any issues, especially not an issue like this..maybe compare the code between dash and ltc electrum and you might spot the problem?

@SomberNight
Copy link
Member

SomberNight commented Dec 17, 2021 •
edited
Loading

@amuna1225 Presumably the reason it does not happen for you with the dash fork is that your wallet there does not have a long history.

There are multiple causes of freezes with large wallets but the primary problem is that upon almost every update the whole list is recreated and redrawn. It is not just the History tab, but also the Addresses and Coins tabs. (Although only the currently visible tab is refreshed on updates, the others are only refreshed on demand when switching to them)

EDIT: To be clear, a viable workaround is to simply create a new wallet/seed and send all your coins there.

@Transisto Transisto mentioned this issue Jan 18, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance 🏎 topic-wallet 👛 related to wallet.py, or maybe address_synchronizer.py/coinchooser.py user-interface 🔲
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@nochiel @Transisto @SomberNight @amuna1225 @ln2max @sureplus1988 @kennedyiowa