Avoid conflicting chain locks when rescan is triggered by new block

[pinheadmz]

Fix for bcoin-org/bcoin#1006 from upstream in bcoin.

Issue summary:

• WalletDB somehow falls behind chain (i.e. botched rescan attempt, maybe due to one of the txdb assertion errors)
• A new block is added to the chain by the network
• The chain block-adding process locks the chain
• WalletDB realizes its behind and initiates a rescan
• Rescan also requires a lock on chain, but chain is still locked and will never unlock until the rescan is done!

Bug fix summary:
Emit the 'block connect' event synchronously so the original 'connect' event from chain can resolve. The chain 'connect' event is emitted asynchronously meaning it waits for all its listeners to resolve before program flow continues. The chain event is proxied by nodeclient to the wallet as 'block connect' and this is what ultimately triggers the reorg.

Note that this is only an issue when the wallet is run as a plugin. If the wallet is being run as a separate node, it gets the 'block connect' event from the node's HTTP websocket:

hsd/lib/node/http.js

Lines 666 to 680 in a1409dc

	this.chain.on('connect', (entry, block, view) => {
	const sockets = this.channel('chain');
	if (!sockets)
	return;
	const raw = entry.encode();
	this.to('chain', 'chain connect', raw);
	for (const socket of sockets) {
	const txs = this.filterBlock(socket, block);
	socket.fire('block connect', raw, txs);
	}
	});

The websocket event is obviously synchronous because it is just a one-way firing, it doesn't wait for a response at all. Indeed! The fact that one mode of operation uses synchronous events should help justify this pull request.

The side effect of this change is that in a few tests we always just expect the walletDB to be up to date with the chain when we add new blocks. This is no longer the case and probably we should always consider wallet and node operations to be asynchronous anyway. For those tests I've added a wait-er to ensure wallet is synced to chain before continuing.

Reviewers can run the new test test/wallet-rescan-test.js after reverting the one-line change in lib/wallet/nodeclient.js and observe that the test will timeout -- this is because the two conflicting locks will wait for each other forever.

[coveralls]

Pull Request Test Coverage Report for Build 497068965

• 1 of 1 (100.0%) changed or added relevant line in 1 file are covered.
• 4 unchanged lines in 3 files lost coverage.
• Overall coverage increased (+0.2%) to 59.805%

---

Files with Coverage Reduction	New Missed Lines	%
lib/mining/miner.js	1	64.58%
lib/net/pool.js	1	32.63%
lib/net/peer.js	2	35.28%

Totals
Change from base Build 480764491:	0.2%
Covered Lines:	19628
Relevant Lines:	30587

---

💛 - Coveralls

[pinheadmz]

Closing for now, although we still may need a fix for this. Turns out we use emitAsync here for a good reason (memory):

bcoin-org/bcoin#932