Retry creating data stream/template when OpenSearch is unreachable #8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Anders Swanson <anders.swanson@oracle.com>
anders-swanson
force-pushed
the
retry-config-connection
branch
from
a234277 to
e8a37d9
Compare
|
@cosmo0920 could you take a look at this? Thank you. |
Signed-off-by: Anders Swanson <anders.swanson@oracle.com>
Signed-off-by: Hiroshi Hatake <cosmo0920.oucc@gmail.com>
cosmo0920
reviewed
Jan 26, 2022
Comment on lines
34
to
43
| delay = 1 | ||
| begin | ||
| @data_stream_names = [@data_stream_name] | ||
| create_index_template(@data_stream_name, @data_stream_template_name, @host) | ||
| create_data_stream(@data_stream_name) | ||
| rescue => e | ||
| raise Fluent::ConfigError, "Failed to create data stream: <#{@data_stream_name}> #{e.message}" | ||
| log.info "Failed to create data stream, will retry in #{delay} second(s): <#{@data_stream_name}> #{e.message}" | ||
| sleep(delay) | ||
| delay *= 2 | ||
| retry |
There was a problem hiding this comment.
Retrying forever is harmful in the current Fluentd core mechanism.
Instead, I've committed another retrying mechanism in 47b4275
There was a problem hiding this comment.
Ok, thank you!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the OpenSearch backend is unreachable when the plugin starts its initial configuration, it will crash the Fluentd container (see #7).
This PR fixes this by adding a retry mechanism to the configuration step, relying on eventual consistency. I did not add a limit to the retry, as the connection should either eventually work, or there is something wrong in the environment that requires manual intervention.