feat: Add exponential backoff when connecting or reconnecting and allow plugin to start without making initial connection

[reimda]

This PR adds the ability to start the kafka_consumer input even if the kafka broker is down. The default behavior of the plugin is to require connection for a successful plugin start, but users can set connection_strategy = "defer" to choose to connect after the plugin has started.

Since starting up without a connection means the plugin will immediately start to retry making a connection, this PR also adds settings to configure retries through sarama's Metadata.Retry.* settings. Previously telegraf always used the sarama defaults of 3 retries and 250ms delay before each retry. Now the Metadata.Retry.* settings are exposed through kafka_consumer's telegraf config so users can choose how many times to retry and configure the retry delay.

The PR also adds exponential backoff of connection retries by implementing a sarama Metadata.Retry.BackoffFunc. Telegraf's default backoff is constant for backward compatibility, but users can set metadata_retry_type = "exponential" to choose exponential backoff.

To test the new settings, I extended the integration test to run with both connection strategies and added two new tests around exponential backoff, one just checking that the BackoffFunc math is right and another that tries to connect to an unopen port on localhost to make sure backoff is configured in sarama correctly

related influxdata/feature-requests#461

[reimda]

Looks like I missed some linter errors.

I'd love it if someone could review the setting names and descriptions in addition to the code. I added the retry settings in common code so they are available for outputs.kafka to use too. I haven't tested them there but I'd expect them to work.

There are other retry timeouts and other settings in sarama (search for retry in https://github.com/Shopify/sarama/blob/main/config.go). I'm relatively sure Metadata.Retry.* are the only ones needed to change connection/reconnection timeouts, but I would love it if anyone could shed more light on what the other ones are for (Admin, Producer, Consumer, Consumer.Group.Rebalance)

[srebhan]

Just a few comments @reimda. Nothing big besides the linter issue.

[srebhan]

Thanks for the update @reimda! Only one question about the warning...

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[srebhan]

Looks good to me. Thanks @reimda for tackling this!