mlogc hangs and/or fails to process data #173
Comments
|
Original reporter: zoroo |
|
brectanus: On my system I get this: $ ps -ef|grep mlogc $ ps -fp15745 What flavor and version of bourne shell is /bin/sh on your system? On mine it is "dash", but some it may be bash, ksh, pdksh, etc. $ ls -lad /bin/*sh You might try linking /bin/sh to bash (or another shell) and see if that helps. |
|
zoroo: if you need another information I can give :) |
ghost
assigned
|
zoroo: $ ps -fp18096 it's good ??? $ ls -lad /bin/*sh bash --version |
|
brectanus: I cannot duplicate this. ModSecurity calls an Apache API function (ap_open_piped_log()) to execute mlogc. So, the bug seems to lie within this API or, more likely, in an underlying APR/APR-Util call for your platform. What apr/apr-util are you using (httpd -V)? I suggest you try the latest Apache httpd with the latest APU/APR-Util and see if this fixes the issue. These are working fine for me: Linux Ubuntu 8.04.1 (hardy) x86_64 with httpd 2.2.9 with either of these sets of apr libs:
If the upgrade does not solve it, then I'll do some more digging. |
|
zoroo: hello, /opt/httpd/bin/httpd -V |
|
brectanus: Have you been able to try a more recent Apache and/or APR/APR-Util to see if this resolves the issue? |
|
brectanus: Dropping priority as I cannot duplicate and am no longer getting feedback. |
|
michaelcaplan: I'm working with a similar environment (centOS 5.2, custom compiled Apache 2.2.10, PHP 5.2.6, mlogc compiled from ModSecurity v2.5.7). I'm seeing mlogc processes growing with each apache start and stop: [root@linux-dev sloccount-2.26]# ps aux | grep mlogc |
|
brectanus: What APR/APR-Util is your Apache instance using (httpd -V)? How was Apache compiled (should be at the top of your config.log where you compiled apache)? Are you using any NFS mounts for the audit logs? |
|
michaelcaplan: # /usr/local/apache2/bin/httpd -V apache was configured as follows: ./configure --enable-so --with-mpm=prefork --enable-ssl --enable-rewrite --enable-unique-id --disable-include --disable-userdir --enable-deflate --with-pcre=/usr No NSF mounts modsecurity configured as follows: ./configure --with-apxs=/usr/local/apache2/bin/apxs --with-lua=/usr/local --with-pcre=/usr --with-apr=/usr/local/apache2 --with-apu=/usr/local/apache2 Anything else I can provide to better diagnose the problem? |
|
brectanus: Thanks. I'll build a setup as you have described and see if I can duplicate it. |
|
brectanus: I built a centos 5.2 box from scratch, built Apache httpd from source and built ModSecurity 2.5.7 from source: $ uname -a $ cat /etc/redhat-release $ /usr/local/apache2/bin/httpd -V Here is what I did: sudo yum install pcre-devel cd apr-1.3.3 cd ../apr-util-1.3.4 cd ../httpd-2.2.10 cd ../modsecurity-apache_2.5.7/apache2 I will attach a tarball of the config I am using (contents of /home/user/httpd_root). This install works just fine for me. The mlogc binary is executed (and stopped) correctly and alerts are sent to the console correctly. I get this after starting httpd: $ sudo /usr/local/apache2/bin/apachectl -f /home/user/httpd_root/conf/httpd.conf -k start (fixed a typo in my original comment) |
|
brectanus: Attached the httpd_root I used in my previous example. Should work if you extract to /home/user. |
|
michaelcaplan: Brian, Thanks for the great work trying to reproduce the problem. In review of your efforts, I discovered that the OS I am having issues with mlogc is not actually centOS 5.2, but 4.7 -- the pains of not keeping dev / staging / production machines in sync. Sorry for the bad information. I went through the same exercise of compiling on a fresh centOS 5.2 box yesterday. Without actually piping content through from apache, mlogc appears to be functioning fine. It is responsive to system signals, which was an issue on the centOS 4.7 box. My configuration on the 5.2 box differs slightly, in that I used the apr and apr-utils libs compiled by apache. I did go back to the 4.7 box and tried to recompile against separately compiled apr and apr-utils libs, as you did. Unfortunately, this had the same effect. mlogc was unresponsive after loading, and could only be terminated with a SIGKILL. I'm going to migrate my 4.7 box to 5.2. It appears that will solve my problems. Thanks, Mike |
|
mark.parsons: Brian, uname -a cat /etc/redhat-release /usr/local/apache-2.2.9/bin/httpd -V here is what I did: sudo yum install pcre-devel cd httpd-2.2.9 cd ../modsecurity-apache_2.5.7/apache2 Configured httpd.conf and modsecurity and mlogc.conf Started apache /usr/local/apache-2.2.9/bin/apachectl start ps-ef|grep 'mlogc|httpd' Looked at where the mlgoc logs are configured to be based on /usr/local/etc/mlogc.confand I only see Contets of mlogc-error.log are: Any help would be very much appreciated. Thanks, |
|
brectanus: It seems to be an issue with APR/APR-Util libs. You are building httpd with included APR, but building modsecurity with the system apr (I believe). Try building with apr/apr-util from source as I did above. tar zxf apr-1.3.3.tar.gz cd apr-1.3.3 cd ../apr-util-1.3.4 Then make sure the httpd build s using the local apr install. Something like this (--with-apr=/usr/local/apr --with-apr-util=/usr/local/apr): cd httpd-2.2.9 |
|
mark.parsons: Brian, I tried what you suggested including a new build of the latest Apache here is what I did: cd apr-1.3.3 add /usr/local/apr/lib to /etc/ld.so.conf cd ../ ldconfig cd httpd-2.2.10 I am still seeing the symptoms I was seeing previously prior to this build. Below is the output of the httpd -V for the new build I did. Server version: Apache/2.2.10 (Unix) thanks, |
|
michaelcaplan: Hi Mark, I too saw no difference in mlogc behavior compiling against the apache http bundled apr and apr-util libs, or seperatly compiled apr and apr-util libs -- on CentOS 4.7. (See my comment above). The only solution found to date (for an unknown reason) was to upgrade to centOS 5 (5.2 specifically) and then compile (either against the apache http bundled apr / apr-util libs or separate). Mike |
|
brectanus: Do you see the same issue using another piped logger with Apache? Try something like this to see if the same thing happens with The =error or access log: ErrorLog "|/bin/cat - > logs/error_log-test" Or even try cat in SecAuditLog: SecAuditLog "|/bin/cat - > logs/modsec_audit-test" Also check out your shell being used (usually /bin/sh, but whatever SHELL_PATH was defined to) and see if it calls fork() when the -c option is used: strace /bin/sh -c "echo foobar" See https://issues.apache.org/bugzilla/show_bug.cgi?id=38989 for why that is bad and what you can do, but you can try changing /bin/sh to be /bin/ksh (or /bin/pdksh) if that shell does not fork. Other than that I am not sure what else it can be as it seems to be RedHat 4.7 and not Apache or ModSecurity. In any case, please update the results here so others can benefit. If it is failing for the error/access logs, then you may want to get with Red Hat support and see if there is a workaround (and post the answer here). Thanks, |
|
brectanus: Mark/Mike, Were either of you able to try my last suggestion? |
|
michaelcaplan: Hi Brian, Sorry for not following up. I've since upgraded my development environment to Centos 5.2 and don't have a 4.7 box at my disposal for testing this out. If there is a need for this type of test, I can do it, but it won't be for about a week. Thanks, Mike |
|
brectanus: I'll leave it up to the others then. I too don't have time to build a 4.7 box at the moment. Thanks. |
|
mark.parsons: Brian, I haven't had much time recently to do any further testing. I did try using SecAuditLog "|/bin/cat - > logs/modsec_audit-test" and I was able to write logs and the system did not hang and the /bin/cat process was stopped as expected when apache was stopped. I did do a strace /bin/sh -c "echo foobar" and the process did not fork. I have not had a chance to call RedHat or try changing the shell that is used. I hope to have time after the Thanksgiving holiday next week to do some testing. Thanks, |
|
pfrederiksen: We are also having this same issue. It works in one environment, and doesn't in another. We have the exact same symptoms. In the environment that it works: Server version: Apache/2.0.52 The environment that it doesn't work: Server version: Apache/2.2.10 (Unix) If I can give anymore information, please let me know. |
|
brectanus: Paul, Please try the /bin/cat tests above and report back the results. This should help to indicate if it is an mlogc issue, a ModSecurity issue, or an Apache/APR issue. Is this RedHat or CentOS? What version? What version of ModSecurity and mlogc? What is the exact line you are using to use mlogc (the SecAuditLog line)? On the first machine you listed, what was the APR/APR-Util version? Was Apache compiled with the APR included with httpd, a system installed APR, or another custom APR install (for both systems)? thanks, |
|
pfrederiksen: 1) Please try the /bin/cat tests above and report back the results. This should help to indicate if it is an mlogc issue, a ModSecurity issue, or an Apache/APR issue.
cat /etc/redhat-releaseFedora release 10.90 (Rawhide)
This is the only entries in the error log no matter what debug level I give:
6)Was Apache compiled with the APR included with httpd, a system installed APR, or another custom APR install (for both systems)? |
|
wayne: Hi, I have the same problem here, except that this appears on a Debian Etch System. Apache 2.2.11 www-data@n2:/$ uname -a www-data@n2:/$ dpkg --list|grep apr Seems like if it's really a mlogc-Issue. mlogc just starts but does nothing actually: n2:/var/log/mlogc# strace -p 5861 n2:/var/log/mlogc# strace -p 5918 n2:/var/log/mlogc# strace -p 5922 After i restart the Apache Webserver, more mlogc-Processes are started: If i cat my audit-Logs by hand, everything is working as expected and the entry is visible in the console: www-data@n2:/$ strace cat /var/log/mlogc/data/index |/usr/local/bin/mlogc /etc/mlogc.conf Ciao, |
|
brectanus: Werner, It looks like mlogc is not running correctly from Apache httpd. It may be that you have compiled Apache httpd with the included libs (apr/apr-util/pcre) but compiled ModSecurity/mlogc with the system libs? What is the output from apachectl -V (httpd -V)? Would you mind attaching your Apache httpd config.log? Or at least the configure line showing all the options you used to compile Apache httpd? Additionally attach the ModSecurity config.log as well. thanks, |
|
wayne: Hi Brian, thanks for your input so far. ln2:~# ldd /usr/local/bin/mlogc Seems like mlogc is picking the Debian libs for apr/apr-util. What i dont understand is the following:
n2:~# /usr/local/bin/mlogc Sure, because it uses the Debian Libs. I've also tried to add /usr/local/apache2/lib to /etc/ld.conf.so, executed ldconfig but mlogc still tries to use the debian libs (also when recompiled) :-( thank your Brian and regards, |
|
wayne: Hi Brian, the Apache HTTPD ist starting mlogc-processes. Here is the requested information: n2:~# httpd -V $ ./configure --with-layout=Apache --enable-ssl --enable-rewrite --enable-expires --enable-headers --disable-cgi --disa ModSecurity config.log will follow. Cheers, |
|
brectanus: It does not work well to use the Apache included libs. You have system wide apr/apu libs installed: www-data@n2:/$ dpkg --list|grep apr But you compiled Apache with the included libs: Server loaded: APR 1.3.3, APR-Util 1.3.4 Mlogc may be picking up the wrong libs when run under apache. What does this give you: ldd /path/to/mlogc Then check further by creating a wrapper script for mlogc (use the wrapper in modsec config) that does this: /path/to/mlogc-wrapper: #!/bin/sh And compare the ldd output from running mlogc-wrapper from the shell vs within apache. You can try removing the Debian installed apr/apr-util packages and see if that helps (you may need to re-configure/re-make modsecurity/mlogc). However what I would do is to keep things consistent and either: Use the debian apr/apr-util packages and specify --with-apr=/usr --with-apr-util=/usr when configuring Apache. OR Compile/install your own apr/apr-util separately if you want the newer versions and then point both Apache and modsecurity to them via --with-* options. NOTE: You may have the same issue with pcre libs. |
|
brectanus: I added these to help diagnose problems like this: MODSEC-59, MODSEC-60 |
|
brectanus: Make sure you run ldconfig to update the cache after updating /etc/ld.so.conf (I think you also have the name of that file wrong above). Did you try setting LD_LIBRARY_PATH=/usr/local/apache2/lib ? Also, make sure you run "make clean" first. The build may be a bit wrong for mlogc and not using the correct versions detected in the ModSecurity configure run. I have updated the build system quite a bit for 2.5.10, so you might want to try the 2.5.x branch versions of configure/Makefile.in: https://mod-security.svn.sourceforge.net/svnroot/mod-security/m2/branches/2.5.x/apache2/ Check the apache2/mlogc-src/Makefile after the configure run and make sure the APR/APU libs are correct. |
|
wayne: Hi Brian,
I've now tried it either way. In the first run, i recompiled the Apache-Webserver with the apr/apu-libs provided by Debian as follows: Subsequent I've recompiled ModSecurity 2.5.9 from scratch - also with the apr/apu-libs provided by Debian as follows: After that I've created mlogc: No luck, mlogc is just not sending anything to the console. If I force it "by hand" (strace cat /var/log/mlogc/data/index |/usr/local/bin/mlogc /etc/mlogc.conf ) everything is working fine. Same result If i use the APR/APU-Libs from the Apache-Source :-( This really drives me nuts :) regards, |
|
brectanus: Attached a patch to add more debugging and fix a locking issue. This patch will apply to 2.5.9. Please apply, re-run "make mlogc", then copy the mlogc binary to its installed location. |
|
nc: Brian's patch fixes the problem here. Thanks. |
|
nc: I'm experiencing this same issue, and have found that mlogc is hanging at the apr_global_mutex_lock(gmutex) call in transaction_log_init(). I'm adding a patch that adds a line of debugging output just prior to this call. When applied, and executed with an ErrorLogLevel of at least 4, a deadlock at this line will result in the mlogc-error.log containing the line: Transaction initialization locking mutex... but not the follow-up line: Transaction initialization started. |
|
nc: Patch adding a line of debugging output to mlogc.c. |
|
brectanus: I just want to verify that when you restart Apache httpd the mlogc processes are terminated correctly (See Michael Caplan's comments above). And, is the parent process now the correct shell/httpd vs '1'? |
|
wayne: ModSecurity Config.log |
|
nc: Yes, stopping httpd stops any mlogc processes. Previously, I think, the signal thread was never started. An mlogc process with a PID of '1' appears briefly at startup and shutdown. Other than those transient appearances, the parent process is httpd. Thanks for the patch. |
|
brectanus: The latest patch seems to solve this issue. Please reopen if this is not the case. |
|
brectanus: If you are seeing this issue on 2.5.9, please apply the patch, re-run "make mlogc", then copy the mlogc binary to its installed location. Apply patch from the main modsecurity-apache_2.5.9 directory as follows: patch -p0 < /path/to/mlogc_2.5.10-dev1.diff |
|
brectanus: Reopening to remove obsolete attachments. |
MODSEC-20: when my httpd server is down :
root@trix:/$ ps -ef | grep mlogc
root 17974 17119 0 18:14 pts/0 00:00:00 grep mlogc
18:14:04 root@trix:/$
after service httpd start
root@trix:/$ps -ef | grep mlogc
root 18097 18096 0 18:15 ? 00:00:00 /opt/mlogc/mlogc /etc/mlogc.conf
root 18099 18096 0 18:15 ? 00:00:00 /opt/mlogc/mlogc /etc/mlogc.conf
root 18122 17119 0 18:15 pts/0 00:00:00 grep mlogc
18:15:50 root@trix$
The text was updated successfully, but these errors were encountered: