Bug: file plugin does not always read files from the beginning dispite start_position=beginning

[fbaligand]

When I configure file input to read all files in a folder from the beginning, I notice that not all files are read from the beginning. For example, some files are read from line 306, all previous lines are not read, and so, lost.
This is really annoying, because we can't rely on this plugin behavior.

The problem seems to happen when there are lots of files (more than 500).
Up to my tests, it is tied to sincedb file size. When it becomes too big, new files that arrive are not read from beginning, sometimes, even not read at all.

If I drop sincedb file, and I restart Logstash, it works fine during some days, until sincedb file becomes big again.

My guess about that bug fix is to remove from sincedb file, files that are older than ignore_older option.

Logstash version : 6.1.3
file input configuration :

	file {
		path => ["/path/to/file-inputs/*.csv"]
		start_position => "beginning"
		codec => plain {
			charset => "UTF-8"
		}
	}

[guyboertje]

Same problem reported in LS repo. elastic/logstash#8929

[guyboertje]

Await feedback on whether sincedb_clean_after setting fixes this problem.

[fbaligand]

Hi @guyboertje,

I will make tests this week with file input version 4.1.0 to check if it fixes the issue.
And obviously, I will give you feedback !

Fabien

[wjq]

Hi @guyboertje ,
sincedb_clean_after is not working for me at current version 6.5.4. Please see my steps:

1. create new sincedb file /tmp/sincedb/sincedb_test1, clear elastic index by curl and make sure it has permission to write
2. update my input as

input {
  file {
    path => "/vagrant/elk_log_test/*.log"
    #mode => "read"
    start_position => "beginning"
    sincedb_clean_after => "20s"
    sincedb_path => "/tmp/sincedb/sincedb_test1"
  }
}

3. run logstash by command:
   bin/logstash -f ~/elk_config/elk_config.conf -w 1 --pipeline.unsafe_shutdown --config.debug --log.level debug
4. after running, create a new apache_access1.log file under elk_log_test folder
5. add one line in log, nothing happened except

[2019-01-10T09:34:17,843][DEBUG][filewatch.tailmode.handlers.grow] read_to_eof: get chunk
[2019-01-10T09:34:17,957][DEBUG][filewatch.sincedbcollection] writing sincedb (delta since last write = 1547112857)

6. add more 4 lines in log, corrected this time, I can see results. and delta since last write is.

[2019-01-10T09:34:39,688][DEBUG][filewatch.sincedbcollection] writing sincedb (delta since last write = 22)

sincedb was written to

123 0 25 1092 1547112879.687172 /vagrant/elk_log_test/apache-access1.log