Fix integration test failures by making Node::with_conf more robust

[jamillambert]

The intermittent integration test failures are caused by race conditions in the creation of the node, wallet or the cookie file. Change Node::with_conf to check if there is a problem and retry the section where it occurred, or if that still fails retry the whole process of starting the node.

First patch:

• Split out some sections into helper functions.

Second patch:

• Remove the loop in with_conf() that creates client, instead loop over the individual parts.
• Create helper functions for creating the client, and creating or loading the client wallet, both with their own retry logic.
• Reduce the sleep time between retries to reduce the delay, since it looks like the errors happen over shorter time periods than the existing 1 second wait.
• Add functions to wait until the client and the cookie file are available.
• Add a loop to retry the whole process if it fails, since the integration tests pass when repeated, just repeat this section where the failures occur instead.
• Remove all the debugging outputs since it all works now they are not needed.
• Update the rustdocs for the function.

Third patch:

• Revert integration_test: Pause after node comes up #206, since the issue is now solved at it’s source the pause is no longer needed.

Closes #205

[tcharding]

Mad, good work man.

[tcharding]

Maybe a code comment here saying why we kill the process if waiting for the cookie file fails?

[jamillambert]

I have added comments to all the checks where it kills the process and retries.

[tcharding]

Perhaps this is cleaner?

        for _ in 0..9 {
            if let Ok(client) = Client::new_with_auth(rpc_url, auth.clone()) {
                return Ok(client);
            }
            thread::sleep(Duration::from_millis(200));
        }
        Client::new_with_auth(rpc_url, auth.clone()).map_err(|e| Error::NoBitcoindInstance(e.to_string()).into())

[jamillambert]

Thanks, it is. I changed it and also rewrote create_client_wallet to use the same format.

[jamillambert]

NB. I think there is a CI issue, this passed CI but I got an error locally when I accidentally ran cargo test in the root instead of integration_test. The failure was a doctest on node/src/lib.rs:205 because I had change the default values but not updated the docs.

[jamillambert]

Made the suggested changes and added a few more comments.

[tcharding]

ACK 8c9a4a4