[tests] fix disconnect_ban intermittency

[jnewbery]

Fixes another functional test race. disconnect_ban.py bans a subnet with a one second expiry and then immediately tests that the subnet is banned. Due to the #10234, calls to listbanned() now result in a disk flush, which can block on busy systems (for example when running multiple functional tests in parallel).

Fix is to use mocktime to control when the ban expires.

[paveljanik]

Sorry, but I was not able to reproduce the failure on my systems, so I can't provide tested ACK.

Out of 1000 runs, I have got 8 of this only:

2017-05-10 08:58:54.014000 TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/private/tmp/bitcoin-master/test/functional/test_framework/util.py", line 281, in start_nodes
    rpcs.append(start_node(i, dirname, extra_args[i], rpchost, timewait=timewait, binary=binary[i]))
  File "/private/tmp/bitcoin-master/test/functional/test_framework/util.py", line 242, in start_node
    wait_for_bitcoind_start(bitcoind_processes[i], url, i)
  File "/private/tmp/bitcoin-master/test/functional/test_framework/util.py", line 219, in wait_for_bitcoind_start
    blocks = rpc.getblockcount()
  File "/private/tmp/bitcoin-master/test/functional/test_framework/coverage.py", line 46, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
  File "/private/tmp/bitcoin-master/test/functional/test_framework/authproxy.py", line 152, in __call__
    response = self._request('POST', self.__url.path, postdata.encode('utf-8'))
  File "/private/tmp/bitcoin-master/test/functional/test_framework/authproxy.py", line 125, in _request
    self.__conn.request(method, path, postdata, headers)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 849, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 712, in create_connection
    raise err
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 703, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/bitcoin-master/test/functional/test_framework/test_framework.py", line 147, in main
    self.setup_network()
  File "/private/tmp/bitcoin-master/test/functional/test_framework/test_framework.py", line 81, in setup_network
    self.setup_nodes()
  File "/private/tmp/bitcoin-master/test/functional/test_framework/test_framework.py", line 94, in setup_nodes
    self.nodes = start_nodes(self.num_nodes, self.options.tmpdir, extra_args)
  File "/private/tmp/bitcoin-master/test/functional/test_framework/util.py", line 283, in start_nodes
    stop_nodes(rpcs)
  File "/private/tmp/bitcoin-master/test/functional/test_framework/util.py", line 303, in stop_nodes
    assert not bitcoind_processes.values() # All connections must be gone now
AssertionError
2017-05-10 08:58:54.017000 TestFramework (INFO): Stopping nodes
Traceback (most recent call last):
  File "./disconnect_ban.py", line 105, in <module>
    DisconnectBanTest().main()
  File "/private/tmp/bitcoin-master/test/functional/test_framework/test_framework.py", line 163, in main
    self.stop_nodes()
  File "/private/tmp/bitcoin-master/test/functional/test_framework/test_framework.py", line 209, in stop_nodes
    stop_nodes(self.nodes)
  File "/private/tmp/bitcoin-master/test/functional/test_framework/util.py", line 301, in stop_nodes
    for i, node in enumerate(nodes):
TypeError: 'NoneType' object is not iterable

Ie. the already known and reported issue #9317 (comment), nothing else.

[jnewbery]

@paveljanik - can you try running the test when you have a lot of disk access happening in parallel which could block disk i/o for the node under test. One way to do this is to run in parallel with pruning.py which does a lot of stop/starts (and therefore flushes to disk a lot). You can do that by changing the BASE_SCRIPTS list in test_runner to something like:

BASE_SCRIPTS= [
    'pruning.py',
    'disconnect_ban.py --portseed=1',
    'disconnect_ban.py --portseed=2',
    'disconnect_ban.py --portseed=3',
    'disconnect_ban.py --portseed=4',
...

[MarcoFalke]

utACK 3ba2c08

[sipa]

Concept ACK

[paveljanik]

I was not able to reproduce, sorry.

But reading the code, utACK 3ba2c08