Implement a 'waiting mode' for cases where RPC calls are not functioning correctly?

[AdamISZ]

See #673 (comment) as a summary of that thread as a whole, the commit which closed that thread did only the "easy" thing of shutting down, and the issue is rather tricky.

I want to strongly emphasize again, that a simple flakiness of network connection should already be handled by extensive Exception catching as well as multiple retries within jsonrpc.py already, this is only for special/unexpected failures.

[AdamISZ]

Indeed, painstaking efforts were made some years ago to try to assure that all error conditions were handled, including timeouts. See:

joinmarket-clientserver/jmclient/jmclient/jsonrpc.py

Lines 150 to 160 in 7f85822

	request = {"method": method, "params": params, "id": currentId}
	#query can fail from keepalive timeout; keep retrying if it does, up
	#to a reasonable limit, then raise (failure to access blockchain
	#is a critical failure). Note that a real failure to connect (e.g.
	#wrong port) is raised in queryHTTP directly.
	response_received = False
	for i in range(100):
	response = self.queryHTTP(request)
	if response != "CONNFAILURE":
	response_received = True
	break

So if that isn't doing what it's intended to do, we, I guess, have a bug we need to fix (or handle an unanticipated case, let's say).

I guess this is one of those things that's significantly harder to do because it has to be reproduced first. Can you get something reproducible (tough one, but if you can that might be our best shot).

[AdamISZ]

Your suggested fix of keeping the retry in blockchaininterface.py certainly doesn't look dumb, but I agree it'd need some careful thought. My first issue is I don't know why the retry in jsonrpc.py itself is not working.

[AdamISZ]

Oh I see. That CONNFAILURE check is only the case of BadStatusLine. So doubtless the "not responding before a timeout" return is different. Maybe we can just check what happens when the other end doesn't respond and put that in jsonrpc.py.

When I originally made this issue I was just vaguely assuming we had the 'connection timed out' case covered, because we had addressed so many cases. But it looks like, maybe not.