Unable to see transactions on imported or newly created wallets

[maxtannahill]

Using version 0.9.2 on macOS 10.15.6 with personal Electrum Rust Server 0.8.5 on Nodl (local IP address for URL, port 50001, no SSL) as well as Blockstream's server (Blockstream.info for URL, port 700 and SSL). Both provide successful connection messages but no transactions visible even after having multiple confirmations.

My Electrum wallet works fine with the server and wallets with the same seed.

BIP39 seeds with m/84'/0'/0' derivation path. Single sig, P2WPKH.

[craigraw]

Finally got a new node up with Electrs 0.8.5 running to test this.

I think this is caused by two issues:

1. Initially I had around 7 wallets open in Sparrow (normal testing conditions). I observed that Electrs was taking up to a minute to retrieve the history for a single address. A full wallet retrieval thus might take much longer. Performance seems to degrade quickly when the server is asked to handle multiple wallets. This is corroborated by @jlopp's Electrum Server Performance Report findings: https://blog.keys.casa/electrum-server-performance-report/

2. I then closed all wallets except for a single BIP39 seed wallet. The performance retrieving address history improved dramatically (although still pretty slow in comparison to ElectrumX). However, Electrs appears to have a bug with notifying that the history of an address has changed - effectively it sends constant unnecessary notifications asking Sparrow to update the wallet history. This likely degraded performance further. A bug report has already been filed on this issue: Incorrect status hash due to incorrect ordering of same-block transactions romanz/electrs#297

It would be possible to Sparrow to improve performance by reducing the number of wallet updates through remembering the invalid Electrs notification calls and ignoring them in future, but ideally this bug needs to be addressed.

[craigraw]

Implemented more efficient Electrum server RPC calls (similar to Electrum's approach) in 4bad46c

Blockstream.info returns now all txes for a fairly dated wallet in a few seconds.

Electrs also works but still has a buggy script hash subscription implementation as noted. While the commit above implements a check that tries to ignore these as far as possible, there are further issues.

Electrs's subscription implementation does not scale well on RPi hardware beyond a single wallet being open, and performance degrades quickly with additional wallets causing random socket closures and other problematic behaviour. In particular, Electrs does not reliably indicate that a script hash has been updated - for example sending a tx between two wallets sometimes sends an update for the first but not the second. In addition, calling get_history on an updated script hash often times out as the CPU core on the RPi is maxed at 100%, so Sparrow's interface cannot update with the new transaction. To make matters worse, Electrs often then unilaterally closes the connection in these situations causing Sparrow to reconnect and resubscribe all script hashes - which does not help the CPU load situation!

After several days of effort trying to resolve these issues it's difficult to recommend Electrs. On limited hardware, it's recommended to increase the wait_duration_secs from 5 seconds to a higher number to reduce the frequency of subscription update scanning, and while this helps it does not resolve the issues described above.

[craigraw]

Added status updates to show wallet loading progress in b7aa73e

[maxtannahill]

Hi Craig, this now seems to be working for me over Tor to my Electrs server. Thank you very much!

[craigraw]

Great! I'm going to close this off now.

If you experience any problems with Electrs, I've opened an issue recently in that repo.