Add option to file source to exclude older files

[fuchsnj]

If you have a single log file that is written to, and rotated periodically (but still matches the include list), it would be nice to allow Vector to not open a file-watcher on the older files. The number of files that are watched can easily exceed the file limit in this scenario.

The ignore_older_secs config option initially seems like it would solve this issue, but this only controls where in a file Vector starts reading from. (If it's older than the setting, it just reads from the end so it only gets new logs). A new setting to completely exclude old files from being watched could be useful.

[vamshivishnu]

Supporting this issue.

Just because when you are file watching log file like Test*.log file when we have too many files on directory like Test-1.log, Test-2.log ... a live file is always a incremented number appended down the road we will have too many Test-xxx.log files in the directory but in theory only 1 live log file. at this case all files are scanned for watch list and getting too many open file issue. Some how Vector need to have knowledge to understand live file only watch as old files have no activity for long time.

[c-hui]

Yeah, I need to close the file handler when a file is not active, avoid too many opened files limit.

[vamshivishnu]

Any update on this @c-hui

[bootswithdefer]

This is impacting me too, I have a syslog server that has thousands of log files and vector keeps files open that it's not reading from.

[vamshivishnu]

@c-hui and @fuchsnj any update ?

[Bingmang]

Meet the same problem, the description of ignore_older_secs is quite confusing.

[Bingmang]

Hi, sorry for bothering, any update on this? @c-hui @fuchsnj

[jszwedko]

Hi, sorry for bothering, any update on this? @c-hui @fuchsnj

No updates yet unfortunately.