Make connections to k8s API server persistent #1390
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrzej-stencel
requested review from
frankreno,
kkujawa-sumo,
perk-sumo,
pmalek-sumo and
sumo-drosiek
as code owners
sumo-drosiek
approved these changes
Jan 29, 2021
pmalek-sumo
approved these changes
Jan 29, 2021
|
I guess you'll be fixing tests now right? |
This should prevent SNAT port usage exhaustion in AKS.
andrzej-stencel
force-pushed
the
use-kubeclient-with-persistent-connections
branch
from
61032f5
to
2339b2f
Compare
sumo-drosiek
reviewed
Feb 1, 2021
fluent-plugin-enhance-k8s-metadata/fluent-plugin-enhance-k8s-metadata.gemspec
Show resolved
Hide resolved
andrzej-stencel
changed the title
andrzej-stencel
deleted the
use-kubeclient-with-persistent-connections
branch
This was referenced Feb 3, 2021
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The Fluentd plugins we use make a good amount of HTTP connections to the Kubernetes API server for Kubernetes metadata enrichment. Until now, the HTTP connections were being closed after each HTTP request, requiring a new HTTP connection to be established on every request to API server. This was a suboptimal solution, and the root cause for SNAT port exhaustion in AKS clusters.
With this change, we are using a yet unreleased version of Kubeclient library to be able to create persistent HTTP connections. The Kubeclient library (which is used by our Fluentd plugins to make requests to k8s API server) originally used RestClient gem under the hood, which does not make it possible to create persistent connections. With the unreleased version of Kubeclient, the underlying HTTP library has been changed from RestClient to Faraday, which, when used with NetHttpPersistent adapter, makes connections persistent by default.
Testing in AKS showed that SNAT port usage with persistent connections is constant, no matter how many requests our plugins make to API server.
Testing performed