Fluentd stuck and stops ingesting logs

[vikramchhibberTPM]

Describe the bug

Hi,

I have forward plugin that received logs and elasticsearch output plugin. In one of our environments, after processing logs for some time, it just gets stuck. We have to restart fluentd server to recover this.

Have attached complete logs for this. Any pointers for this issue are appreciated.
logs.txt

Thanks

To Reproduce

Not sure

Expected behavior

Even if there is elasticsearch connectivity issue, fluentd should recover once the connectivity is restored and should start processing the logs.

Your Environment

- Fluentd version:
- TD Agent version:
td-agent 4.4.2 fluentd 1.15.3 (e89092ce1132a933c12bb23fe8c9323c07ca81f5)
- Operating system:
debian:buster Docker base image
- Kernel version:

Your Configuration

<match debug.**>
  @type stdout
  @id output_stdout
</match>

####
## Source descriptions:
##

# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
# @see http://docs.fluentd.org/articles/in_http
<source>
  @type http
  @id input_http
  port {{ or (env "INPUT_HTTP_PORT") "8888" }}
</source>

## live debugging agent
<source>
  @type debug_agent
  @id input_debug_agent
  bind 127.0.0.1
  port 24230
</source>

# FOO - BEGIN
<source>
  @type monitor_agent
  bind 0.0.0.0
  port 9999
</source>

# Comment this to disable debug mode
<system>
  @log_level {{ or (env "LOG_LEVEL") "warn" }}
</system>

<source>
  @type forward
  port {{ or (env "INPUT_SYSLOG_TCP_PORT") "9899" }}
  bind {{ or (env "INPUT_SYSLOG_TCP_BIND_IP") "0.0.0.0" }}
  tag syslogs.edge.v2
</source>

<match syslogs.edge.v2>
  @type rewrite_tag_filter
  <rule>
    key FOO_ENTERPRISE_ID
    pattern ^(.+)$
    tag ${tag}.$1
  </rule>
</match>

<match syslogs.edge.v2.**>
  @type elasticsearch
  @log_level {{ or (env "LOG_LEVEL") "warn" }}
  # Replace this by Elasticsearch IP address
  host {{ or (env "ELASTICSEARCH_HOST") "localhost" }}
  port {{ or (env "ELASTICSEARCH_PORT") "9200" }}
{{ if env "ELASTICSEARCH_USERNAME" }}
  user {{ env "ELASTICSEARCH_USERNAME" }}
  password {{ env "ELASTICSEARCH_PASSWORD" }}
{{ end }}
{{ if env "ELASTICSEARCH_SCHEME" }}
  scheme {{ env "ELASTICSEARCH_SCHEME" }}
{{ end }}
  # If Elasticsearch is in cluster, use one of the following format
  # hosts host1:port1,host2:port2,host3:port3
  # hosts https://customhost.com:443/path,https://username:password@host-failover.com:443
  reconnect_on_error true
  logstash_format true
  logstash_prefix ${tag}
  request_timeout 15s
  <buffer tag>
    @type memory
    # Logs flush interval
    flush_interval 5s
    flush_thread_count 4s
    overflow_action block
  </buffer>
  template_overwrite true
  template_name syslogs_edge_index_template_v4
  template_file /fluentd/etc/template.json
</match>

# FOO - END

Your Error Log

Please check the attachment.

Additional context

No response

[mmcoralogix]

Hello Team,
Any findings on above.

[daipom]

Some retries succeeded, but some keeps failing.
I don't know what the difference is.

2023-06-04 16:00:32 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2023-06-04 16:00:33 +0000 chunk="5fd4fde00d4b2a4e13cd69b101650ee1" error_class=Elastic::Transport::Transport::Error error="no address for gcs-production-ec-main.es.eastus.azure.elastic-cloud.com (Resolv::ResolvError)"
2023-06-04 16:00:32 +0000 [warn]: #0 retry succeeded. chunk_id="5fd4fe273b95bc97c4cab7e4802a1462"
2023-06-04 16:25:34 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2023-06-04 16:25:36 +0000 chunk="5fd503b1871033bff85b47a34795f1aa" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"gcs-production-ec-main.es.eastus.azure.elastic-cloud.com\", :port=>9243, :scheme=>\"https\", :user=>\"gcsclient\", :password=>\"obfuscated\"}): connect_write timeout reached"
2023-06-04 16:25:35 +0000 [warn]: #0 failed to flush the buffer. retry_times=1 next_retry_time=2023-06-04 16:25:38 +0000 chunk="5fd503b292291c62e89a6e9592361d90" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 16:25:35 +0000 [warn]: #0 failed to flush the buffer. retry_times=1 next_retry_time=2023-06-04 16:25:38 +0000 chunk="5fd503b31ba711b12e8a5da7a07b1517" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 16:25:35 +0000 [warn]: #0 retry succeeded. chunk_id="5fd503b1871033bff85b47a34795f1aa"
2023-06-04 16:25:36 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2023-06-04 16:25:38 +0000 chunk="5fd503b38a85df48460a5cc048ed174d" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 16:25:37 +0000 [warn]: #0 retry succeeded. chunk_id="5fd503b38a85df48460a5cc048ed174d"
2023-06-04 17:30:55 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2023-06-04 17:30:57 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:30:55 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2023-06-04 17:30:57 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:30:57 +0000 [warn]: #0 failed to flush the buffer. retry_times=1 next_retry_time=2023-06-04 17:30:59 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:30:57 +0000 [warn]: #0 failed to flush the buffer. retry_times=1 next_retry_time=2023-06-04 17:31:00 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:11 +0000 [warn]: #0 failed to flush the buffer. retry_times=2 next_retry_time=2023-06-04 17:31:16 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:11 +0000 [warn]: #0 failed to flush the buffer. retry_times=2 next_retry_time=2023-06-04 17:31:16 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:14 +0000 [warn]: #0 failed to flush the buffer. retry_times=2 next_retry_time=2023-06-04 17:31:19 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:14 +0000 [warn]: #0 failed to flush the buffer. retry_times=2 next_retry_time=2023-06-04 17:31:19 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:33 +0000 [warn]: #0 failed to flush the buffer. retry_times=3 next_retry_time=2023-06-04 17:31:43 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:33 +0000 [warn]: #0 failed to flush the buffer. retry_times=3 next_retry_time=2023-06-04 17:31:43 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:33 +0000 [warn]: #0 failed to flush the buffer. retry_times=3 next_retry_time=2023-06-04 17:31:41 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:33 +0000 [warn]: #0 failed to flush the buffer. retry_times=3 next_retry_time=2023-06-04 17:31:43 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:57 +0000 [warn]: #0 failed to flush the buffer. retry_times=4 next_retry_time=2023-06-04 17:32:14 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:57 +0000 [warn]: #0 failed to flush the buffer. retry_times=4 next_retry_time=2023-06-04 17:32:14 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:57 +0000 [warn]: #0 failed to flush the buffer. retry_times=4 next_retry_time=2023-06-04 17:32:15 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:31:57 +0000 [warn]: #0 failed to flush the buffer. retry_times=4 next_retry_time=2023-06-04 17:32:14 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:32:28 +0000 [warn]: #0 failed to flush the buffer. retry_times=5 next_retry_time=2023-06-04 17:33:02 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:32:28 +0000 [warn]: #0 failed to flush the buffer. retry_times=5 next_retry_time=2023-06-04 17:33:05 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:32:29 +0000 [warn]: #0 failed to flush the buffer. retry_times=5 next_retry_time=2023-06-04 17:33:04 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:32:30 +0000 [warn]: #0 failed to flush the buffer. retry_times=5 next_retry_time=2023-06-04 17:32:59 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:33:14 +0000 [warn]: #0 failed to flush the buffer. retry_times=6 next_retry_time=2023-06-04 17:34:16 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:33:16 +0000 [warn]: #0 failed to flush the buffer. retry_times=6 next_retry_time=2023-06-04 17:34:23 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:33:19 +0000 [warn]: #0 failed to flush the buffer. retry_times=6 next_retry_time=2023-06-04 17:34:19 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:33:19 +0000 [warn]: #0 failed to flush the buffer. retry_times=6 next_retry_time=2023-06-04 17:34:28 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:34:42 +0000 [warn]: #0 failed to flush the buffer. retry_times=7 next_retry_time=2023-06-04 17:36:40 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:34:42 +0000 [warn]: #0 failed to flush the buffer. retry_times=7 next_retry_time=2023-06-04 17:36:50 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:34:42 +0000 [warn]: #0 failed to flush the buffer. retry_times=7 next_retry_time=2023-06-04 17:36:51 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:34:42 +0000 [warn]: #0 failed to flush the buffer. retry_times=7 next_retry_time=2023-06-04 17:36:40 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:36:54 +0000 [warn]: #0 failed to flush the buffer. retry_times=8 next_retry_time=2023-06-04 17:40:41 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:36:54 +0000 [warn]: #0 failed to flush the buffer. retry_times=8 next_retry_time=2023-06-04 17:40:45 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:37:05 +0000 [warn]: #0 failed to flush the buffer. retry_times=8 next_retry_time=2023-06-04 17:41:28 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:37:05 +0000 [warn]: #0 failed to flush the buffer. retry_times=8 next_retry_time=2023-06-04 17:41:42 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:41:57 +0000 [warn]: #0 failed to flush the buffer. retry_times=9 next_retry_time=2023-06-04 17:51:20 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:41:57 +0000 [warn]: #0 failed to flush the buffer. retry_times=9 next_retry_time=2023-06-04 17:49:56 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:41:57 +0000 [warn]: #0 failed to flush the buffer. retry_times=9 next_retry_time=2023-06-04 17:51:17 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:41:57 +0000 [warn]: #0 failed to flush the buffer. retry_times=9 next_retry_time=2023-06-04 17:51:16 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:51:30 +0000 [warn]: #0 failed to flush the buffer. retry_times=10 next_retry_time=2023-06-04 18:07:30 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:51:30 +0000 [warn]: #0 failed to flush the buffer. retry_times=10 next_retry_time=2023-06-04 18:08:14 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:51:32 +0000 [warn]: #0 failed to flush the buffer. retry_times=10 next_retry_time=2023-06-04 18:07:45 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 17:51:35 +0000 [warn]: #0 failed to flush the buffer. retry_times=10 next_retry_time=2023-06-04 18:10:45 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 18:11:00 +0000 [warn]: #0 failed to flush the buffer. retry_times=11 next_retry_time=2023-06-04 18:45:58 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 18:11:00 +0000 [warn]: #0 failed to flush the buffer. retry_times=11 next_retry_time=2023-06-04 18:44:46 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 18:11:00 +0000 [warn]: #0 failed to flush the buffer. retry_times=11 next_retry_time=2023-06-04 18:48:16 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 18:11:00 +0000 [warn]: #0 failed to flush the buffer. retry_times=11 next_retry_time=2023-06-04 18:41:16 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 18:41:30 +0000 [warn]: #0 failed to flush the buffer. retry_times=12 next_retry_time=2023-06-04 19:53:49 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 19:54:03 +0000 [warn]: #0 failed to flush the buffer. retry_times=13 next_retry_time=2023-06-04 22:27:27 +0000 chunk="5fd5124cfb9337058aec85dfd2123a84" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 19:54:03 +0000 [warn]: #0 failed to flush the buffer. retry_times=13 next_retry_time=2023-06-04 22:08:45 +0000 chunk="5fd5124ef4c2819952989a17f76cdee1" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 19:54:03 +0000 [warn]: #0 failed to flush the buffer. retry_times=13 next_retry_time=2023-06-04 21:54:38 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 19:54:03 +0000 [warn]: #0 failed to flush the buffer. retry_times=13 next_retry_time=2023-06-04 22:06:32 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 22:06:47 +0000 [warn]: #0 failed to flush the buffer. retry_times=14 next_retry_time=2023-06-05 02:48:10 +0000 chunk="5fd5124ef4f1ff5231ecc70cf18e3189" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"
2023-06-04 22:06:47 +0000 [warn]: #0 failed to flush the buffer. retry_times=14 next_retry_time=2023-06-05 03:11:56 +0000 chunk="5fd5124d9f99735fe2d8ceda9d3c6024" error_class=Elastic::Transport::Transport::Error error="connect_write timeout reached"

[daipom]

It looks like the issue of fluent-plugin-elasticsearch.
If you can't find the cause, you can ask it there.

[daipom]

To Reproduce

Not sure

With this information and the logs, we can't judge if it is a bug or not.
So I move this issue to discussions.

• Contributing Guidelines