Add connectivity tests on startup to the actual instances

[hilts-vaughan]

Some customers are confused by "Ready for new connections" and then try and connect and find it fails. This is often a routing problem but some of them incorrectly assume that the message above means everything was validated and fine.

Should we perform some kind of connectivity test when creating the instances or is this too much?

[kurtisvg]

I think we should definitely provide this feature. It probably should be toggle-able by a flag, and I'm unsure if it should be enabled by default. I'd be open to other folks opinions on the matter.

[hilts-vaughan]

I think unless all the examples show the flag it won't see widespread adoption and many users will still confuse themselves. I would be inclined for default but it may not technically be backwards compatible if for some reason on startup the proxy can't connect (but presumably it is later able to)

[tmaiaroto]

Yes please! I'm experience an issue with this now. Randomly my container is failing with OAuth and finally got into a broken container today to look around. There's an issue with SSL certs. Yet will continue along saying "Ready for new connections" ... Which it's not. While I'd really love to know why sometimes it's unable to make any HTTPS requests, in the least a failure would help a lot. It could re-deploy a working container.

[hilts-vaughan]

Hey Kurtis,

In the case where we detect failed connectivity on instances, what should we do? What if there are multiple instances only one of them fail? Crash? Emit a warning?

[kurtisvg]

As a developer, I would find it most useful if it failed and emitted a reason why.

[hilts-vaughan]

Maybe we could implement this as just a basic TCP connectivity test, since that would let us remain agnostic of the actual underlying protocols?

This would rule out routing and firewall problems, but of course not catch EVERYTHING that could go wrong.

[tmaiaroto]

Not being able to connect to the OAuth endpoint is a pretty big thing to go wrong though. Would love to see it fail should that happen. If I can get the time, I may take a look and see if I can add a pull request or something.

[hilts-vaughan]

I think you're right -- we should try and catch that too. Though, let's fry one fish at a time. :)

[enocom]

Related to #137.

[enocom]

This would be an easy feature to add now that we have a CheckConnections method on the internal proxy struct. We currently use this when health checks are enabled. It would be easy to allow callers to opt-in to a connection check on proxy startup.