New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fail2ban broken on CentOS 6 using inotify #44
Comments
@leeclemens Could you have a look at this issue? Sounds like a big blow and we definitely needs to fix it up one way or another |
Quick fix for this, for users, is to use
to switch back to gamin for file change monitoring. |
Interesting scenerio, sorry it's been so long to reply. To clarify, the issue is with the logfile fail2ban is monitoring is rotated? Using inotify, new log is not detected (and subsequently not monitored); however, using gamin the new log is detected? Can you include your logrotate.d configuration for the log file being rotated? |
@bgrooren bah, that's not a fix, that's a workaround :) may ultimately be an issue with pyinotify or ultimately how you are rotating the log - so please do include that information. Can you also include the version of pyinotify you are using? |
That's why I said "quick fix" ;-) I'm using stock centos 6, which has python-inotify-0.9.1-1.el6.noarch. The default logrotate config for sshd is (/etc/logrotate.d/syslog):
Like I mentioned before, the reason for inotify not working is that centos 6 uses the following defaults (/etc/logrotate.conf):
So: "create" instead of "copyrotate". Inotify tracks changes by inode, so when the logfile opened at fail2ban start time is rotated, no more changes are written to it. Fail2ban fails to detect the fact that the file was rotated, and does not start tracking changes to new newly created logfile. |
The default in RHEL is create as well (since 5, at least). If you enable debug logging, do you see something like this after the secure log has been rotated and then written to?
|
Hi, I've just enabled debug logging on one of the affected servers. As soon as the secure log is rotated I will post the result here. |
Ok so last night a log rotation happened; this is what /var/log/messages says:
So somehow it does pick up the log rotation of the exim (mailserver) log, but it does not log anything regarding the secure log. |
Hi :) I have some expirience with pyinotify and the problem ur talking about is 100% 'cose ur not catchin IN_CREATE event. You shoud catch it in, 'cose the rotation most time is : 1.Move file to *.1 2. Create new file .
|
I'm seeing exactly the same problem and have confirmed that the I'm a huge fan of fail2ban and recommend it to all of my clients. Thanks! |
someone should look into taking the fix @l4m3rx suggested ... I started to craft a unittest for that but haven't had a chance to finish it... may be in upcoming week ... |
I can confirm this problem on 5 CentOS 6.2 servers. As soon as my logs rotated Sunday just after midnight, fail2ban stopped banning. Logs show "fail2ban.filter : INFO Log rotation detected" which was my first clue that it had something to do with it. |
ok -- I have pushed a _enh/test_backends branch into my clone which supposed to fix this issue + provides unittests for polling and pyinotify (disabled for gamin since it seems completely ... not working atm for me). Due to all the threading and tests being as close to real use cases as possible, tests are still somewhat fragile and fail from time to time ;) but pyinotify now should work -- it will monitor parent directory for CREATE event. Please test if you get a chance : |
ok -- that point was not yet good ;) now (0.8.6-95-gc0c1232) that branch seems to work just perfect. If I hear no complaints or do not see problem with my instance -- I will merge it into master tomorrow, thus closing this issue |
…nds, unittests * _enh/test_backends: RF: reordered tests + enabled gamin now that its fix is pending in Debian ENH+BF: filtergamin -- to be more inline with current design of filterinotify ENH: 1 more sleep_4_poll to guarantee difference in time stamp ENH: few more delays for cases relying on time stamps ENH: tests much more robust now across pythons 2.4 -- 2.7 BF+RF: pyinotify refreshes watcher upon CREATE, unified/simplified *(add|del)LogPath among *Filters ENH: fail2ban-testcases -- custom logging format to ease debugging, non-0 exit code in case of failure ENH: Filter's testcases -- rename, del + list again --- a bit unstable, might still fail from time to time BF: pyinotify -- monitor the parent directory for IN_CREATE + process freshly added file (Closes gh-44) ENH: first working unittest for checking polling and inotify backends RF/BF: just use pyinotify.ThreadedNotifier thread in filterpyinotify RF: filter.py -- single readline in a loop ENH: FilterPoll -- adjusted some msgs + allowed to operate without jail (for testing) Minor additional comment to DEVELOP ENH: extended test LogfileMonitor
* commit '0.8.6-90-g08564bd': ENH: fail2ban-testcases -- custom logging format to ease debugging, non-0 exit code in case of failure ENH: Filter's testcases -- rename, del + list again --- a bit unstable, might still fail from time to time BF: pyinotify -- monitor the parent directory for IN_CREATE + process freshly added file (Closes fail2bangh-44) ENH: first working unittest for checking polling and inotify backends RF/BF: just use pyinotify.ThreadedNotifier thread in filterpyinotify RF: filter.py -- single readline in a loop ENH: FilterPoll -- adjusted some msgs + allowed to operate without jail (for testing) Minor additional comment to DEVELOP ENH: extended test LogfileMonitor ENH: add more verbosity levels to be controlled while running unittests Added few tests of FileFilter. yet to place them into a Jail-ed execution test DOC: distilling some of server "design" into DEVELOP notes for common good ENH: minor, just trailing spaces/tabs + reformated a string ENH: added a basic test for FilterPoll for detection of modifications clarified that the are existing test cases and the 'coming soon' is about creating new ones. Added beginnings of documentation for developers BF: usedns=no was not working at all RF: filtertestcase.py to put common testing into a helping subroutine ENH: be able to control verbosity from cmdline for fail2ban-testcases
I am still seeing this in gentoo net-analyzer/fail2ban-0.8.7.1. I will switch DEBUG on. In the meantime here is the logrotate of the one with the problem. /var/log/yate /var/log/yate-cdr.tsv { |
What python version? If less than 2.6 try current master |
Python 2.7.3-r2 and 3.2.3
|
then more information would be needed (indeed output with debug level), and please also run fail2ban-testcases -- do all of them pass? would you know a quick way to install centos in a chroot? e.g. in debian land we have debootstrap allowing to do that in 1 command... may be I could replicate the issue locally |
disregard the last one -- you are using gentoo, although original report about centos -- so that is what confused me... might be worth starting a new bug report I would still advise trying current master though |
fail2ban-testcases is not installed. I will download it from github. I seem to be able to install debootstrap on my gentoo but I would first try
|
weurope trash # git clone https://github.com/fail2ban/fail2ban.git
|
After the rotation no mention of the /var/log/yate since today (2012/11/24) Let me know if there is something else I could try. For the time being I will switct back to pooling which seems to work OK |
On Sat, 24 Nov 2012, octavsly wrote:
do you have multiple jails pointing to the same log? Yaroslav O. Halchenko |
Only once[yate] enabled = true /var/log/yate /var/log/yate-cdr.tsv { |
This is still not working with fail2ban 0.8.8 and python-inotify 0.9.1 on EL6: |
I should note that not only does the test fail, it fails in production after logrotation. |
I am also experiencing this failure. CentOS6 (2.6.32-279.19.1.el6.i686) Fail2ban stopped banning after the installation of pyinotify which occurred as a dependency when I updated the fail2ban package to 0.8.8 from the EPEL repository. Fail2ban worked fine previously. For the moment I am putting a "[DEFAULT] backend=gamin" into the jail.local and it works again. Cheers, |
have the same failure I tried changing the jail.conf to backend = gamin restarted sshd / fail2ban secure log is still empty and the system is still logging into the rotated log /var/log/secure until i restarted rsyslogd |
Hello all, I using Centos 6.4 and have problems with fail2ban. When i start fail2ban it says ok. Logs give no errors. Testing using /usr/bin/python2.6 ..........F.................FFFFFF .....FAIL: testInaccessibleFile (testcases.clientreadertestcase.ConfigReaderTest)Traceback (most recent call last): FAIL: testNewChangeViaIsModified (testcases.filtertestcase.LogFileMonitor)Traceback (most recent call last): FAIL: test_delLogPath (testcases.filtertestcase.MonitorFailures(/tmp/monitorfailures_FilterPollCNpLbdfail2ban))Traceback (most recent call last): FAIL: test_grow_file (testcases.filtertestcase.MonitorFailures(/tmp/monitorfailures_FilterPollCNpLbdfail2ban))Traceback (most recent call last): FAIL: test_move_file (testcases.filtertestcase.MonitorFailures(/tmp/monitorfailures_FilterPollCNpLbdfail2ban))Traceback (most recent call last): FAIL: test_move_into_file (testcases.filtertestcase.MonitorFailures(/tmp/monitorfailures_FilterPollCNpLbdfail2ban))Traceback (most recent call last): FAIL: test_move_into_file_after_removed (testcases.filtertestcase.MonitorFailures(/tmp/monitorfailures_FilterPollCNpLbdfail2ban))Traceback (most recent call last): FAIL: test_new_bogus_file (testcases.filtertestcase.MonitorFailures(/tmp/monitorfailures_FilterPollCNpLbdfail2ban))Traceback (most recent call last): FAIL: test_rewrite_file (testcases.filtertestcase.MonitorFailures(/tmp/monitorfailures_FilterPollCNpLbdfail2ban))Traceback (most recent call last): Ran 100 tests in 186.577s FAILED (failures=9) What can this be and what can i do to let run fail2ban correct? |
ha -- doesn't it look like what I just reported from a sparc box?
|
Very strange... yesterday a kernel update for Centos 6.4 and some more (32MB) in total. After update reboot my VPS, everything was started. Did a manual startup of Fail2ban and it is working again.. It blocks, it send emails that it has block something... ehmm :) but i'm happy again |
Hi,
I see some weird behavior on some of our CentOS (5.x and 6.x) boxes running fail2ban (from EPEL).
After some investigation it turns out that out-of-the-box it fails to handle the log rotation of /var/log/secure (sshd logs).
So I took a look at my logs and the fail2ban sourcecode, and noticed the following:
By default CentOS does not use "copytruncate" while rotating /var/log/secure, but "create". The result is that the "full" logfile is moved and a new file with the original name is created. This applies to both types of servers.
Looking at the source code of fail2ban I see that in server/filter.py (class FileContainer) it checks for a rotated log file as indicated by a different first line or inode number.
Next, I see that FilterGamin listens for GAMCreated, GAMChanged and GAMExists.
FilterPyinotify listens for IN_MODIFY;
Now not being an expert on Gamin and Inotify I assume this is why it is no longer working:
A simple solution appears to be to also monitor for IN_MOVED_SELF, but according to https://github.com/manos/python-inotify-tail_example this does not work with pyinotify on files.
The text was updated successfully, but these errors were encountered: