Healthchecks should use the same http client

[lukesteensen]

Ref #2234

Currently, many of our healthchecks instantiate their own http client. This opens up the possibility for mismatched configs, etc.

We should go through each sink and ensure that, as much as possible, things like http clients are created in build and used for both the healthcheck and the sink itself.
For non-http-based sinks, we should at least double check that all connection settings are propogated correctly and look for ways to consolidate them.

Todo

• Construct tcp healthcheck with TLS.
  • Affects sinks: vector, datadog, papertrail, tcp, socket.

• Build HttpClient once.
  • All of our http sinks now depend on our util so this is actually quite straight forward.
  • Affects sinks: splunk_hec, loki, logdna, http, ...

• Build aws clients once.
  • Affects all aws sinks.

[bruceg]

Did #2238 really close this? From my reading it only changed how TLS settings were handled, not the management of the HTTP client.

[LucioFranco]

You're right #2238 doesn't directly solve this though, I do not think its as worth it. The problem for me lies more so with the complexness of building our tls types. I think we should ensure TlsSettings can only be built from the correct TlsConfig type instead of doing what was fixed in #2238.

From going through the sinks I have not really found it to be that helpful to build a single httpclient for both the sink and healthcheck. The things that build the httpclient are what we should ensure are the same.

That's mostly what I think but I'll reopen this since we may want to fix this in the future. But our current TLS issues are solved.

[LucioFranco]

So I am shifting this issue to something that is more so good to have rather than an actual bug since the bug this was originally connected has been solved in #2238.

[ktff]

Now all current sinks either have:

• Client created once in/during build method
• No healthcheck
• Client like thing but requires await so it isn't practical to have it in build, but is created by a single method
• Sink and healthcheck use different client like things. ex. kafka