[electrs] unable to sync, failed to get confirmed status

[sandipndev]

Description

After generating many txs (100+) to some given address, syncing with electrs node fails with erratic errors.

❯ bdk-cli -n regtest wallet -w test --server tcp://127.0.0.1:60401 --descriptor "wpkh($XPUB)" sync
[2021-07-23T16:43:47Z ERROR bdk_cli] Electrum(Protocol(Object({"code": Number(1), "message": String("failed to get confirmed status")})))

❯ bdk-cli -n regtest wallet -w test --server tcp://127.0.0.1:60401 --descriptor "wpkh($XPUB)" sync
[2021-07-23T16:45:56Z ERROR bdk_cli] Electrum(AllAttemptsErrored([JSON(Error("invalid type: null, expected a sequence", line: 0, column: 0)), JSON(Error("invalid type: null, expected a sequence", line: 0, column: 0)), JSON(Error("invalid type: null, expected a sequence", line: 0, column: 0)), JSON(Error("invalid type: null, expected a sequence", line: 0, column: 0)), JSON(Error("invalid type: null, expected a sequence", line: 0, column: 0))]))

I would like to add that when I tested using 5-10 transactions, it worked properly, only when the amount of transactions is big, the electrs server reports querying would take a long time and sync fails eventually.

Logs

When I was requesting these syncs, I captured the section of logs generated from the docker container above which outputs logs straight from electrs via this, here are the logs exactly at the moment:

INFO - [172.17.0.1:48184] connected peer
WARN - rpc #1 blockchain.scripthash.get_history [String("06994e480bd40d53eab72d8711bea9a6c16ae64f6f196f6af0e3ae888c8c58ff")] failed: Error: failed to get confirmed status
Caused by: 101+ transactions found, query may take a long time

DEBUG - [172.17.0.1:48184] shutting down connection
INFO - [172.17.0.1:48184] disconnected peer
INFO - [172.17.0.1:48190] connected peer
WARN - rpc #0 blockchain.scripthash.get_history [String("06994e480bd40d53eab72d8711bea9a6c16ae64f6f196f6af0e3ae888c8c58ff")] failed: Error: failed to get confirmed status
Caused by: 101+ transactions found, query may take a long time

DEBUG - [172.17.0.1:48190] shutting down connection
INFO - [172.17.0.1:48190] disconnected peer
INFO - [172.17.0.1:48196] connected peer
WARN - rpc #0 blockchain.scripthash.get_history [String("06994e480bd40d53eab72d8711bea9a6c16ae64f6f196f6af0e3ae888c8c58ff")] failed: Error: failed to get confirmed status
Caused by: 101+ transactions found, query may take a long time

DEBUG - [172.17.0.1:48196] shutting down connection
INFO - [172.17.0.1:48196] disconnected peer

Environment

I am using bitcoindevkit/electrs:0.4.0 docker image which uses a local regtest network for testing. Required ports are exposed following the instructions here. Also, I am using bdk version BDK CLI 0.2.1-dev.

Reproduction steps

# Electrum setup
alias elstart='docker run --detach --rm -p 127.0.0.1:18443-18444:18443-18444/tcp -p 127.0.0.1:60401:60401/tcp --name electrs bitcoindevkit/electrs:0.4.0'
alias elcli='docker exec -it electrs /root/bitcoin-cli -regtest -datadir=/root/.bitcoin $@'

# Starting electrs locally
elstart

# Setting XPUB
export XPUB="[e60cffb9/84'/1'/0'/0]tpubDEx5YjF96AHoeZcL7ay2LFJfzj4BX6avNKYsUT1krBGHB2vwyYMAH3zH64ySeFSRnuHJ6iH3kPPixDgzXK918zxipTeh5iy8TVu8n3sbGQy/*"

# Syncing
bdk-cli -n regtest wallet -w test --server tcp://127.0.0.1:60401 --descriptor "wpkh($XPUB)" sync

# Checking balance
bdk-cli -n regtest wallet -w test --server tcp://127.0.0.1:60401 --descriptor "wpkh($XPUB)" get_balance

# Getting some amount of money to this address (generated from the above XPUB)
elcli -regtest generatetoaddress 101 bcrt1qmdwsk29xjjk3pm4xekzacdvlnqvhd28mncw2rc

# Resync - here's where we get the error
bdk-cli -n regtest wallet -w test --server tcp://127.0.0.1:60401 --descriptor "wpkh($XPUB)" sync

Expected Behaviour

I think syncs should not depend on the number of txs present in the blockchain. This problem could probably be an issue with timeouts during sync.

[sandipndev]

The linked PR fixes this issue in regtest mode for devs running electrum locally.

Otherwise, this would depend on whether or not there is throttling on --txid-limit flag of the electrs instance the bdk library is connected to. Since from bdk we probably don't have any control of electrs, ideally, this issue should be detected and an appropriate error must be thrown.

[notmandatory]

I've also updated the bitcoindevkit/electrs:latest image on docker hub with this fix.

https://hub.docker.com/layers/bitcoindevkit/electrs/latest/images/sha256-f74fa94ff886cd74e9da5423c073fefaa61aed0e70c337b2e06d32c2a08d0031?context=explore