FileBeat5.3.0 ERR Failed to publish events caused by: write tcp 10.6.50.242:33416->10.6.30.70:7044: write: connection reset by peer

[Yeading]

2018-01-22T17:23:44+08:00 ERR Failed to publish events caused by: write tcp 10.6.84.24:60396->10.6.30.71:6044: write: connection reset by peer
2018-01-22T17:23:44+08:00 INFO Error publishing events (retrying): write tcp 10.6.84.24:60396->10.6.30.71:6044: write: connection reset by peer
2018-01-22T17:23:54+08:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=927 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=1 libbeat.logstash.published_but_not_acked_events=1 libbeat.publisher.published_events=1 publish.events=1 registrar.states.update=1 registrar.writes=1
2018-01-22T17:24:24+08:00 INFO No non-zero metrics in the last 30s

[Yeading]

Logstash can receive the data from filebeat:
[2018-01-22T18:03:46,263][DEBUG][logstash.pipeline ] filter received {"event"=>{"@timestamp"=>2018-01-22T10:03:45.165Z, "offset"=>42032, "@Version"=>"1", "input_type"=>"log", "beat"=>{"hostname"=>"10-6-50-242-.jhydesn.idc", "name"=>"10.6.50.242", "version"=>"5.5.2"}, "host"=>"10-6-50-242-.jhydesn.idc", "source"=>"/var/log/topsango_log/login/login.log", "message"=>"2018-01-22 18:03:45,112|INFO|Login|localhost-startStop-1|| - ################## Server node LOGIN@10.6.50.242 start ok. #####################", "type"=>"log", "fields"=>{"area"=>"Domestic", "gameid"=>213, "server"=>"LoginS.s1", "gtype"=>"mg", "hostip"=>"10.6.50.242", "gname"=>"e21e2"}, "tags"=>["beats_input_codec_plain_applied"]}}
[2018-01-22T18:03:46,268][DEBUG][logstash.filters.mutate ] filters/LogStash::Filters::Mutate: removing field {:field=>"message"}
[2018-01-22T18:03:46,269][DEBUG][logstash.pipeline ] output received {"event"=>{"msg_s"=>"localhost-startStop-1|| - {dataSource-6} inited", "offset"=>35892, "source"=>"/var/log/topsango_log/login/login.log", "type"=>"log", "source_file"=>"2018-01-22", "tags"=>["beats_input_codec_plain_applied", "_grokparsefailure"], "duration"=>2018-01-22T10:03:40.102Z, "@timestamp"=>2018-01-22T10:03:40.162Z, "cat"=>"INFO", "@Version"=>"1", "host"=>"10-6-50-242-.jhydesn.idc", "module_s"=>"DruidDataSource", "fields"=>{"gameid"=>213, "area"=>"Domestic", "server"=>"LoginS.s1", "gtype"=>"mg", "hostip"=>"10.6.50.242", "gname"=>"s21s2"}}}

[Yeading]

Just filebeat has the error

[ph]

@Yeading We had issue in the past with the logstash-input-beats that was causing the server to not send the keep alive back to the client, not sending the keep would cause connection reset by peer on the FB side.

Can you add the following to this issue:

• logstash version
• Logstash-input-beats version (using the bin/logstash-plugin list --verbose beats command)

after added this to the issue I think you should update the plugin using the following command:

bin/logstash-plugin update logstash-input-beats

[Yeading]

logstash version：logstash-5.5.2
[root@10-6-30-70-sslj logstash-5.5.2-m36ji-7044]# bin/logstash-plugin list --verbose beats
logstash-input-beats (3.1.23)

I found that logstash haven't receive the data from the Filebeat,In fact.
Thank you very much for helping me.

[Yeading]

[root@10-6-30-70-sslj logstash-5.5.2-m36ji-7044]# bin/logstash-plugin update logstash-input-beats
Updating logstash-input-beats
Updated logstash-codec-multiline 3.0.6 to 3.0.9
Updated logstash-codec-plain 3.0.3 to 3.0.6
Updated logstash-input-beats 3.1.23 to 3.1.26

[Yeading]

After update the logstash-input-beats to 3.1.26,
logstash can receive the data like ,but can't write into the es

[Yeading]

I don't know why

[manish2479]

I am using Logstash v6.2.2.1 with file beat plugin logstash-input-beats (5.0.10) and still I am getting connection reset error.

2018-03-06T06:25:08.137Z        DEBUG   [harvester]     log/log.go:85   End of file reached: /var/log/messages; Backoff now.
2018-03-06T06:25:09.138Z        DEBUG   [harvester]     log/log.go:85   End of file reached: /var/log/messages; Backoff now.
2018-03-06T06:25:09.138Z        DEBUG   [transport]     transport/client.go:201 handle error: write tcp 192.168.25.253:37032->192.168.26.43:5044: write: connection reset by peer
2018-03-06T06:25:09.138Z        DEBUG   [logstash]      logstash/async.go:142   8 events out of 8 events sent to logstash host 192.168.26.43:5044. Continue sending
2018-03-06T06:25:09.138Z        DEBUG   [logstash]      logstash/async.go:99    close connection
2018-03-06T06:25:09.139Z        DEBUG   [transport]     transport/client.go:114 closing
2018-03-06T06:25:09.139Z        ERROR   logstash/async.go:235   Failed to publish events caused by: write tcp 192.168.25.253:37032->192.168.26.43:5044: write: connection reset by peer
2018-03-06T06:25:09.139Z        DEBUG   [logstash]      logstash/async.go:99    close connection
2018-03-06T06:25:10.139Z        ERROR   pipeline/output.go:92   Failed to publish events: write tcp 192.168.25.253:37032->192.168.26.43:5044: write: connection reset by peer
2018-03-06T06:25:10.139Z        DEBUG   [logstash]      logstash/async.go:94    connect
2018-03-06T06:25:11.138Z        DEBUG   [harvester]     log/log.go:85   End of file reached: /var/log/messages; Backoff now.
2018-03-06T06:25:12.139Z        ERROR   pipeline/output.go:74   Failed to connect: dial tcp 192.168.26.43:5044: getsockopt: connection refused
2018-03-06T06:25:12.140Z        DEBUG   [logstash]      logstash/async.go:94    connect
2018-03-06T06:25:13.169Z        DEBUG   [prospector]    prospector/prospector.go:124    Run prospector
2018-03-06T06:25:13.169Z        DEBUG   [prospector]    log/prospector.go:147   Start next scan
2018-03-06T06:25:13.169Z        DEBUG   [prospector]    log/prospector.go:361   Check file for harvesting: /var/log/messages
2018-03-06T06:25:13.169Z        DEBUG   [prospector]    log/prospector.go:447   Update existing file for harvesting: /var/log/messages, offset: 426796
2018-03-06T06:25:13.169Z        DEBUG   [prospector]    log/prospector.go:499   Harvester for file is still running: /var/log/messages

I have also tried Filebeat client timeout configuration as "client_inactivity_timeout => 86400"on the logstash server but still seeing same issues.

[manish2479]

Hi

This seems to be the known issues with logstash elasticsearch plugin and when the configuration is having stdout using rubydebug this issue appears and it fails.

stdout {
  codec => rubydebug
}

To get rid of this either it needs to be patched or stop using stdout as rubydebug. Here is the patch which I applied and that seems fixed my issues.

[https://github.com/awesome-print/awesome_print/pull/324/files]