Retry network join on startup

[ThetaSinner]

Summary

I was initially thinking that kitsune should take responsibility for this but the join operation is important and does need to succeed before the app should be considered active. For that to work though, kitsune would need to understand when it is 'online' and take responsibility for retrying bootstrapping each agent, then communicating that back to the conductor. That mixes responsibilities and seems more complicated than this.

There are two drawbacks to this approach though

• The retryable check is a bit brittle. At the moment the first required network access is a call to the bootstrap server over HTTP which results in a reqwest error. I think this could be addressed by having Kitsune communicate its online/offline state instead of checking the error that's coming back.
• Holochain doesn't finish starting until it has a network connection. This isn't 'bad' necessarily but it's a behavior change

TODO:

• CHANGELOG(s) updated with appropriate info
• Just before pressing the merge button, ensure new entries to CHANGELOG(s) are still under the UNRELEASED heading

[jost-s]

LGTM

[jost-s]

e.is_connect funny name for a method

[ThetaSinner]

It is, it's trying to say "is_a_connection_error" but has been shortened right down to about the least characters possible while still making sense

[maackle]

Looks good, but concerned about a tight, potentially infinite loop

[maackle]

Congrats, I think you're the first person to write code using the funky status state machine other than me :)

[maackle]

Will this cause an infinite loop of retries if the network can't be joined though? Perhaps there needs to be an exponential backoff or at least delay?

[ThetaSinner]

A bit tricky this. Because connecting to the bootstrap server has a timeout, this isn't a fast loop. It will retry as fast as the timeout on the bootstrap connect. So at the moment it's a nice balance between retry rate and responsiveness when the internet comes back on. However, a small change elsewhere to the network initialization would cause a problem here.

We could limit the retries or start with a low delay and slowly increase it just to have some control here. What do you think?

[maackle]

What's the bootstrap connect timeout?

[ThetaSinner]

There are multiple timeouts involved it seems but experimentally it's retrying every 17 second for me. That's a lot of tries if the bootstrap server can't be reached for several hours or more. It's a nice short wait when the server is available again.

[maackle]

I guess that's OK, right? Shouldn't be huge overhead for offline users. I don't think we have bandwidth to do anything more sophisticated like detecting the absence and presence of internet connection, and waiting any longer than 15-30 seconds feels like too long.

[ThetaSinner]

Agreed on both, I think dropping into a network connectivity check with a back-off would be a better solution but not a small bit of work. There is a connectivity check starting in the tx5 repo but it doesn't cover any of the things I think we'd want here like checking for an active network interface, a local gateway and connectivity to the bootstrap server.

[ThetaSinner]

This should be an initial fix for #1580, though the behavior for offline can still be improved