fix: Re-use TCP connections for Kafka dials

[Praveenrajmani]

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

The connections should be properly closed on kafka target pings

Motivation and Context

Fixes #18857

How to test this PR?

• Configure bucket notifications + kafka
• Trigger events
• Check netstat if there are too many ESTABLISHED connections without being closed

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Unit tests added/updated
• Internal documentation updated
• Create a documentation update request here

[Praveenrajmani]

We can also assign a single connection per broker and reuse it instead of doing net.Dial each time. Will fix.

[mctoohey]

Thanks @Praveenrajmani for the fix!
I just tried out this change. Looks good! I no longer see hundreds of new connections being opened and the number of failures sending to Kafka due to this issue has gone to zero.