Transactions being returned out of order

[0xarvin]

Via BDK, we're pulling transactions for a Wallet through a signet electrum instance. Recently, we've noticed that the order that we call batch_transaction_get_raw isn't being respected by electrum and results in the transactions being returned by the client to also be out of order as the reader takes the top result if it's unable to grab a hold of the lock on the reader

This doesn't affect production calls in BDK but only in tests as there's a debug assert here. Was wondering if there's anything that I'm missing or if this is a known issue.

Wallet Descriptor for example:
wsh(sortedmulti(2,[e5cd860e/84'/1'/0']tpubDFVZtuMvX7n6DCWbNAgq361KiJdVFBy4D1sZmdafF1zDNW6dv79MwbK6JBRxGV42BS4STayPsG7Prn3QT7LQwzfaBhE8YhgCs4VzhkYVWTg/0/*,[73ee041b/84'/1'/0']tpubDEPCQp4KHW3sS5o18QbknuC6dtQJPFgWtS6e8wm6ewnc2B1vucCnKf9uf9gCddHeJoK5d5hgJv5ji5Q8VnpT4h9ydsi9FE4t8F2GoPrFPYa/0/*,[08a73043/84'/1'/0']tpubDEZ565bfePfqSVkSfrLaR7g5nruLDWxJFKkJvKWUpMEQBsS9p2f5um5uTwgmKWuHbyoiythmEvDRSgdEoYBgLXfosBsjMBTp1jev9wfi2q9/0/*

Network: Signet

[afilini]

I don't think this was a known issue, and I'm guessing this may be related to the specific server your are running because I've never seen this happen with electrs.

Looking at the code it's also very weird that this is happening: we do have code to re-order the replies from the server for batch requests:

rust-electrum-client/src/raw_client.rs

Lines 706 to 723 in ca48628

	for req_id in missing_responses.iter() {
	match self.recv(&receiver, *req_id) {
	Ok(mut resp) => answers.insert(req_id, resp["result"].take()),
	Err(e) => {
	// In case of error our sender could still be left in the map, depending on where
	// the error happened. Just in case, try to remove it here
	warn!("got error for req_id {}: {:?}", req_id, e);
	warn!("removing all waiting req of this batch");
	let mut guard = self.waiting_map.lock()?;
	for req_id in missing_responses.iter() {
	guard.remove(req_id);
	}
	return Err(e);
	}
	};
	}
	Ok(answers.into_iter().map(|(_, r)| r).collect())

Either there's an issue there, or the server is replying with the wrong request id. Can you try running your tests with trace log level for this crate and then attaching the log here? From that log we should be able to see if the server replies are correct.

[0xarvin]

That's what I'm starting to think, but here's the trace. I replaced the electrum server but let me know if I should include that.

https://gist.github.com/0xarvin/9baee245a45e6e409d9be65e4a447101

[notmandatory]

Which electrum server software are you using where you see this issue?

[0xarvin]

@notmandatory We spun up a set of signet nodes:

url: ssl://electrum.nodes.wallet.build:51002
electrum server: fulcrum

[conduition]

+1 for this issue. Happens for many servers, not just one special guy. Most servers I tested exhibit this bug. Including:

ssl://fulcrum.grey.pw:51002
ssl://electrum.jochen-hoenicke.de:50099
tcp://v7gtzf7nua6hdmb2wtqaqioqmesdb4xrlly4zwr7bvayxv2bpg665pqd.onion:50001
tcp://venmrle3xuwkgkd42wg7f735l6cghst3sdfa3w3ryib2rochfhld6lid.onion:50001
ssl://blackie.c3-soft.com:57002
ssl://e.keff.org:50002
tcp://udfpzbte2hommnvag5f3qlouqkhvp3xybhlus2yvfeqdwlhjroe4bbyd.onion:60001
tcp://kittycp2gatrqhlwpmbczk5rblw62enrpo2rzwtkfrrr27hq435d4vid.onion:50001
ssl://horsey.cryptocowboys.net:50002

I can confirm the cause does NOT originate within this library.

The culprit electrum servers are not respecting JSON-RPC request IDs correctly. If i ask for a series of transactions with a given set of request IDs, the server will return responses for those request IDs but the actual response data might not be mapped to the correct response IDs.

E.g. request with ID 9 asks for TX abcd..., but the response with ID 9 instead contains TX beef... which was requested elsewhere in the batch under a different request ID.

I imagine this is a bug somewhere in the upstream servers' async request handling code. We could easily fix this inside this library by sorting the response TXs by their true TXID. However, if the servers aren't respecting JSON-RPC request IDs, it's hard to know what other properties of JSON-RPC they might be breaking. I didn't test any other batch methods to see if they suffered from the same problem as batch_transaction_get. I think more testing is required here.

The fact that so many servers are apparently broken in this respect is concerning and suggests that maybe a client-side hack is needed, otherwise callers will inevitably reopen issues like this in the future.

[conduition]

I can confirm that this problem is not limited to batched transaction fetching. It also affects batch_script_get_history. If I had to guess, i suspect afflicted servers just don't respect JSON-RPC request IDs for any methods.