add simple sniffer for simple proxy/lb cases #459
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cosmo0920
reviewed
Aug 10, 2018
| @@ -86,6 +86,8 @@ def initialize(retry_stream) | |||
| config_param :pipeline, :string, :default => nil | |||
| config_param :with_transporter_log, :bool, :default => false | |||
| config_param :emit_error_for_missing_id, :bool, :default => false | |||
| config_param :sniffer_class_name, :string, :default => nil | |||
| config_param :reload_after, :integer, :default => -1 | |||
There was a problem hiding this comment.
Could you add a new constant for reload_after and use it?
DEFAULT_RELOAD_AFTER = -1
#...
config_param :reload_after, :integer, :default => DEFAULT_RELOAD_AFTER
# ...
if local_reload_connections && @reload_after > DEFAULT_RELOAD_AFTERis better.
When going through a proxy or load balancer to Elasticsearch, do not try to use the _nodes discovery mechanism - instead, just assume the given host(s) are the only hostnames to use. This is useful e.g. in the case of Kubernetes going through an Elasticsearch Service, where you want to periodically reconnect the long-lived http connections to evenly spread the load throughout the cluster.
richm
force-pushed
the
v0.12-simple-sniffer
branch
from
1ffe376
to
7a126c7
Compare
|
LGTM |
cosmo0920
added a commit
that referenced
this pull request
Aug 12, 2018
add simple sniffer for simple proxy/lb cases Co-Authored-By: Rich Megginson <rmeggins@redhat.com>
cosmo0920
added a commit
that referenced
this pull request
Aug 12, 2018
add simple sniffer for simple proxy/lb cases Co-Authored-By: Rich Megginson <rmeggins@redhat.com>
cosmo0920
added a commit
that referenced
this pull request
Aug 13, 2018
Merge pull request #459 from richm/v0.12-simple-sniffer
richm
added a commit
to richm/origin-aggregated-logging
that referenced
this pull request
Aug 14, 2018
…ing-mux or elasticsearch to help balance sessions https://bugzilla.redhat.com/show_bug.cgi?id=1489533 uken/fluent-plugin-elasticsearch#459 implements support for reloading connections when the Elasticsearch is behind a proxy/load balancer, as in our case, and allows specifying the reload interval in terms of the number of operations. This PR adds support for the following env. vars which can be set in the fluentd daemonset/mux deployment. The ability to set these is provided primarily for experimentation, not something which will ordinarily require tuning in production. `ES_RELOAD_CONNECTIONS` - boolean - default `true` `ES_RELOAD_AFTER` - integer - default `100` `ES_SNIFFER_CLASS_NAME` - string - default `Fluent::Plugin::ElasticsearchSimpleSniffer` There are also `OPS_` named env. vars which will override the corresponding `ES_` named env. var. That is, by default, fluentd will reload connections to Elasticsearch every 100 operations (NOTE: not every 100 records!) These include internal `ping` operations, so will not exactly correspond to each bulk index request.
richm
added a commit
to richm/origin-aggregated-logging
that referenced
this pull request
Aug 14, 2018
…ing-mux or elasticsearch to help balance sessions https://bugzilla.redhat.com/show_bug.cgi?id=1489533 uken/fluent-plugin-elasticsearch#459 implements support for reloading connections when the Elasticsearch is behind a proxy/load balancer, as in our case, and allows specifying the reload interval in terms of the number of operations. This PR adds support for the following env. vars which can be set in the fluentd daemonset/mux deployment. The ability to set these is provided primarily for experimentation, not something which will ordinarily require tuning in production. `ES_RELOAD_CONNECTIONS` - boolean - default `true` `ES_RELOAD_AFTER` - integer - default `100` `ES_SNIFFER_CLASS_NAME` - string - default `Fluent::Plugin::ElasticsearchSimpleSniffer` There are also `OPS_` named env. vars which will override the corresponding `ES_` named env. var. That is, by default, fluentd will reload connections to Elasticsearch every 100 operations (NOTE: not every 100 records!) These include internal `ping` operations, so will not exactly correspond to each bulk index request.
openshift-cherrypick-robot
pushed a commit
to openshift-cherrypick-robot/origin-aggregated-logging
that referenced
this pull request
Aug 15, 2018
…ing-mux or elasticsearch to help balance sessions https://bugzilla.redhat.com/show_bug.cgi?id=1489533 uken/fluent-plugin-elasticsearch#459 implements support for reloading connections when the Elasticsearch is behind a proxy/load balancer, as in our case, and allows specifying the reload interval in terms of the number of operations. This PR adds support for the following env. vars which can be set in the fluentd daemonset/mux deployment. The ability to set these is provided primarily for experimentation, not something which will ordinarily require tuning in production. `ES_RELOAD_CONNECTIONS` - boolean - default `true` `ES_RELOAD_AFTER` - integer - default `100` `ES_SNIFFER_CLASS_NAME` - string - default `Fluent::Plugin::ElasticsearchSimpleSniffer` There are also `OPS_` named env. vars which will override the corresponding `ES_` named env. var. That is, by default, fluentd will reload connections to Elasticsearch every 100 operations (NOTE: not every 100 records!) These include internal `ping` operations, so will not exactly correspond to each bulk index request.
richm
added a commit
to richm/origin-aggregated-logging
that referenced
this pull request
Aug 15, 2018
…ing-mux or elasticsearch to help balance sessions 3.9 bz - https://bugzilla.redhat.com/show_bug.cgi?id=1616354 uken/fluent-plugin-elasticsearch#459 implements support for reloading connections when the Elasticsearch is behind a proxy/load balancer, as in our case, and allows specifying the reload interval in terms of the number of operations. This PR adds support for the following env. vars which can be set in the fluentd daemonset/mux deployment. The ability to set these is provided primarily for experimentation, not something which will ordinarily require tuning in production. `ES_RELOAD_CONNECTIONS` - boolean - default `true` `ES_RELOAD_AFTER` - integer - default `100` `ES_SNIFFER_CLASS_NAME` - string - default `Fluent::Plugin::ElasticsearchSimpleSniffer` There are also `OPS_` named env. vars which will override the corresponding `ES_` named env. var. That is, by default, fluentd will reload connections to Elasticsearch every 100 operations (NOTE: not every 100 records!) These include internal `ping` operations, so will not exactly correspond to each bulk index request. (cherry picked from commit f084fee)
This was referenced Jul 7, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When going through a proxy or load balancer to Elasticsearch,
do not try to use the _nodes discovery mechanism - instead, just
assume the given host(s) are the only hostnames to use.
This is useful e.g. in the case of Kubernetes going through an
Elasticsearch Service, where you want to periodically reconnect
the long-lived http connections to evenly spread the load throughout
the cluster.
(check all that apply)
versionin gemspec are untouchedelasticsearch_dynamic(not required but recommended)