If OpenSearch backend is not ready, FD exits when trying to create Data Stream #7
Description
(check apply)
- read the contribution guideline
- (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.
Problem
Fluentd container exits because OpenSearch is unreachable when plugin initializes. If the connection to OpenSearch takes some time to be ready, Fluentd will enter a CrashLoopBackoff state because of the container restarts. See container log below.
...
Steps to replicate
Start the plugin such that authorization will not work. In my case we were routing auth via a proxy to Keycloak, and Keycloak was not fully initialized.
Expected Behavior or What you need to ask
The plugin will retry the connection, expecting eventual consistency. The plugin should not cause the entire container to crash
if it cannot authenticate to OpenSearch during startup.
...
Using Fluentd and ES plugin versions
- OS version 1.1
- Bare Metal or within Docker or Kubernetes or others? KiND cluster
- Fluentd v0.12 or v0.14/v1.0
- paste result of
fluentd --versionortd-agent --version
fluentd 1.12.3
- paste result of
- ES plugin 3.x.y/2.x.y or 1.x.y
- paste boot log of fluentd or td-agent
2022-01-19 17:47:12 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluentd-standalone.conf"
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.3'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.6.0'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-oci-logging' version '1.0.2'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.0'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.0'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.8.5'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-01-19 17:47:12 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.4'
2022-01-19 17:47:12 +0000 [info]: gem 'fluentd' version '1.12.3'
The client is unable to verify distribution due to security privileges on the server side. Some functionality may not be compatible if the server is running an unsupported product.
2022-01-19 17:47:14 +0000 [error]: config error file="/fluentd/etc/fluentd-standalone.conf" error_class=Fluent::ConfigError error="Failed to create data stream: <verrazzano-system> [401] 401 Unauthorized\n"
- paste result of
fluent-gem list,td-agent-gem listor your Gemfile.lock
*** LOCAL GEMS ***
addressable (2.7.0)
aws-eventstream (1.2.0)
aws-partitions (1.547.0)
aws-sdk-core (3.125.3)
aws-sigv4 (1.4.0)
bigdecimal (default: 1.4.1)
bundler (2.2.6, default: 1.17.2)
cmath (default: 1.0.0)
concurrent-ruby (1.1.8)
cool.io (1.7.1)
csv (default: 3.0.9)
date (default: 2.0.0)
domain_name (0.5.20190701)
e2mmap (default: 0.1.0)
elasticsearch (7.12.0)
elasticsearch-api (7.12.0)
elasticsearch-transport (7.12.0)
elasticsearch-xpack (7.12.0)
etc (default: 1.0.1)
excon (0.80.1)
faraday (1.9.3, 1.4.1)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.3)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.1.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
ffi (1.15.0)
ffi-compiler (1.0.1)
fiddle (default: 1.0.0)
fileutils (default: 1.1.0)
fluent-config-regexp-type (1.0.0)
fluent-plugin-concat (2.4.0)
fluent-plugin-dedot_filter (1.0.0)
fluent-plugin-detect-exceptions (0.0.13)
fluent-plugin-elasticsearch (5.0.3)
fluent-plugin-grok-parser (2.6.2)
fluent-plugin-json-in-json-2 (1.0.2)
fluent-plugin-kubernetes_metadata_filter (2.6.0)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-oci-logging (1.0.2)
fluent-plugin-opensearch (1.0.0)
fluent-plugin-parser-cri (0.1.0)
fluent-plugin-prometheus (1.8.5)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-systemd (1.0.4)
fluentd (1.12.3)
forwardable (default: 1.2.0)
http (4.4.1)
http-accept (1.7.0)
http-cookie (1.0.3)
http-form_data (2.3.0)
http-parser (1.2.3)
http_parser.rb (0.6.0)
inifile (3.0.0)
io-console (default: 0.4.7)
ipaddr (default: 1.2.2)
irb (default: 1.0.0)
jmespath (1.5.0)
json (2.6.1, default: 2.1.0)
jsonpath (1.1.0)
jwt (2.3.0)
kubeclient (4.9.1)
logger (default: 1.3.0)
lru_redux (1.1.0)
matrix (default: 0.1.0)
mime-types (3.3.1)
mime-types-data (3.2021.0225)
msgpack (1.4.2)
multi_json (1.15.0)
multipart-post (2.1.1)
mutex_m (default: 0.1.0)
netrc (0.11.0)
oci (2.15.0)
oj (3.11.0)
opensearch-api (1.0.0)
opensearch-ruby (1.0.0)
opensearch-transport (1.0.0)
openssl (default: 2.1.2)
ostruct (default: 0.1.0)
prime (default: 0.1.0)
prometheus-client (0.9.0)
psych (default: 3.1.0)
public_suffix (4.0.6)
quantile (0.2.1)
rake (13.0.3)
rdoc (default: 6.1.2)
recursive-open-struct (1.1.3)
rest-client (2.1.0)
retriable (3.1.2)
rexml (default: 3.1.9)
rss (default: 0.2.7)
ruby2_keywords (0.0.4)
scanf (default: 1.0.0)
sdbm (default: 1.0.0)
serverengine (2.2.3)
shell (default: 0.7)
sigdump (0.2.4)
stringio (default: 0.0.2)
strptime (0.2.5)
strscan (default: 1.0.0)
sync (default: 0.5.0)
systemd-journal (1.4.2)
thwait (default: 0.1.0)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2021.1)
unf (0.1.4)
unf_ext (0.0.7.7)
webrick (1.7.0, default: 1.4.2)
yajl-ruby (1.4.1)
zlib (default: 1.0.0)
- ES version (optional)
- ES template(s) (optional)