opensearch_data_stream mandates host to be configured

[ayaranga]

(check apply)

• read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Problem

opensearch_data_stream mandates host to be configured even though hosts is configured. Otherwise following error is seen in the logs.

[error]: config error file="/fluentd/etc/fluentd-standalone.conf" error_class=Fluent::ConfigError error="Failed to create data stream: <data-stream-name> Connection refused - connect(2) for 127.0.0.1:9200 (Errno::ECONNREFUSED)"

This is due to

fluent-plugin-opensearch/lib/fluent/plugin/out_opensearch_data_stream.rb

Line 36 in 32ff1ad

create_index_template(@data_stream_name, @data_stream_template_name, @host)

and

fluent-plugin-opensearch/lib/fluent/plugin/out_opensearch_data_stream.rb

Line 164 in 32ff1ad

create_index_template(data_stream_name, data_stream_template_name, host)

passes the host configuration which is set to localhost by default.

Steps to replicate

Do not configure host for opensearch_data_stream
Example:

     @type opensearch_data_stream
     @id out_systemd
     @log_level info
     log_es_400_reason true

     data_stream_name datas-stream-name
     data_stream_template_name data-stream-template

     hosts "#{ENV['ELASTICSEARCH_URL']}"
     ca_file "#{ENV['CA_FILE']}"
     ...

Expected Behavior or What you need to ask

When hosts is configured, it should not expect host also to be configured.

Using Fluentd and ES plugin versions

• OS version NA
• Bare Metal or within Docker or Kubernetes or others? NA
• Fluentd v0.12 or v0.14/v1.0
  • fluentd --version
    • fluentd 1.12.3
• ES plugin 3.x.y/2.x.y or 1.x.y
  • fluent-plugin-opensearch 1.0.0

[cosmo0920]

We don't recommend to use hosts parameter on out_opensearch and out_opensearch_stream plugins.
This parameter causes serious connection problem for https scheme when using user and password authentication.

[ayaranga]

@cosmo0920 Thanks for your response. hosts alone in out_opensearch works fine but not in out_opensearch_data_stream.

• When we use only hosts in out_opensearch_data_stream, as mentioned above, the following error is seen:

[error]: config error file="/fluentd/etc/fluentd-standalone.conf" error_class=Fluent::ConfigError error="Failed to create data stream: <data-stream-name> Connection refused - connect(2) for 127.0.0.1:9200 (Errno::ECONNREFUSED)"

• When we use only host in out_opensearch_data_stream with complete url (Example: http://:), the following error is seen in the logs:

2022-01-19 04:59:48 +0000 [info]: gem 'fluentd' version '1.12.3'
2022-01-19 04:59:52 +0000 [error]: config error file="/fluentd/etc/fluentd-standalone.conf" error_class=Fluent::ConfigError error="Failed to create data stream: <data-stream-name> no address for http (Resolv::ResolvError)"

Specifying host, port and scheme separately worked. Doesn't host config parameter support urls?

[cosmo0920]

host does not handle complete URL.
scheme parameter should handle http or https part.
You should use just hostname in host parameter.

[cosmo0920]

How about the patch #9?
This patch works for your configuration?

[cosmo0920]

Merged #9. Closing.

[ayaranga]

@cosmo0920 Sorry for the delay in response. I just got a chance to test your fix. The data stream creation and template going through fine by just configuring hosts parameter. However it is failing to push the logs to OpenSearch. Following error is seen in the log file:

2022-02-08 08:28:27 +0000 [warn]: #0 [out_all] failed to flush the buffer. retry_time=6 next_retry_seconds=2022-02-08 08:28:37 +0000 chunk="5d77d7e1bf72fa2deb4513c66ac65a16" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (<datastream-name>): Connection refused - connect(2) for 127.0.0.1:9200 (Errno::ECONNREFUSED)"

This is due to

fluent-plugin-opensearch/lib/fluent/plugin/out_opensearch_data_stream.rb

Line 172 in 4649e09

host = @host

[cosmo0920]

That line is already removed at the latest fluent-plugin-opensearch source tree. Could you try it again?

[anders-swanson]

see: #21