Option to limit initial file scanning

[morphles]

Parsing/scaning large log files is a bit problematic, say files >500MB take some time and CPU power to be parsed. Also modyfing log format to drop info and thus make file smaller just to run fail2ban does not sound optimal/desiarable. It would seem that very often, most of log file is not that relevant. Well at least in my use case, probably days worth of logging, at most, is what is relevant. Though I could even live with no "pre-scanning" of log file. So it would be nice if jail could define something like:
log_tail =
If that is specified it should only look at log lines on startup. Additionall alternative (though considerably more complicated to implement):
log_scan_only_last_min =
If that is specified only, scan lines that have timestamps that are less than num minutes before now. Obviously scanning would need to happen backwards in such case.

Or at least option to disable "pre-scaning" miget even be enough. Though obviously this (well all mentioned options) should be on per jail basis.

[sebres]

Option log_tail is imho not required, because since v.0.9 the last offset of log file will be saved in database and after restart, f2b will parse file after this position only (if not new file).
Suggested option log_scan_only_last_min is indirect already included - it's findtime.

But current implementation of filter makes a full scan (to find a time now - findtime), if we have a new log file / initial state (new jail enabled). In addition it have a little bug (see #795 (comment)).

I have developed a new algorithm (half-interval search) for very fast seek to now-findtime within my major branch #716 - see sebres@53a30a2 and sebres@c1637e9
It can be cheery-picked into master branch together with bug fix above.
@fail2ban/contributors should I do it?

[szepeviktor]

Called binary search.

[sebres]

@szepeviktor know-it-all :)
No, I know this under "half-interval search algorithm" - but see the first para inside your link...

[sebres]

Conclusion:

• use logrotate
• since database introduced, the last position of the log file (saved in database) will be used after restart of service (recognize logrotate etc. with checking of md5 of the first line in the log file)
• specify option "tail" after name of logfile in jail.local (for seek to end of file)
• specify not so large findtime, since 0.10 the log-line corresponding findtime will be found using fast search algorithm (from last saved position)