Apache children segfaulting using Event MPM

[GoogleCodeExporter]

>> What version of the product are you using (please check X-Mod-Pagespeed
header)?

Tried the current beta RPM for CentOS and tried to build the module from source 
(0.9.1.1-r171).

>> On what operating system?

CentOS 5.5 (Final)

>> Which version of Apache?

Apache 2.2.17 (latest version of Apache installed via EasyApache/cPanel)

>> Which MPM?

Event

>> Please provide any additional information below, especially a URL or an
HTML file that exhibits the problem.

I was showing the following error in the Apache error logs after enabling the 
module.

*** glibc detected *** /usr/local/apache/bin/httpd: double free or corruption 
(fasttop): 0x0e5f6510 ***
======= Backtrace: =========
/lib/libc.so.6[0x4965a5]
/lib/libc.so.6(cfree+0x59)[0x4969e9]
/usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x7ec581]
/usr/local/apache/modules/mod_pagespeed.so[0x20b4c3]
/usr/local/apache/modules/mod_pagespeed.so[0x20b5a1]
/usr/local/apache/modules/mod_pagespeed.so[0x20bcd5]
/usr/local/apache/modules/mod_pagespeed.so[0x2074dc]
/usr/local/apache/bin/httpd(ap_run_handler+0x46)[0x8084259]
/usr/local/apache/bin/httpd(ap_invoke_handler+0x14d)[0x80849bc]
/usr/local/apache/bin/httpd(ap_process_request+0x63)[0x80f34ff]
/usr/local/apache/bin/httpd[0x80efdfc]
/usr/local/apache/bin/httpd(ap_run_process_connection+0x46)[0x808c9f7]
/usr/local/apache/bin/httpd[0x8131b0c]
/usr/local/apache/bin/httpd[0x813286a]
/usr/local/apache/lib/libapr-1.so.0[0x661438]
/lib/libpthread.so.0[0x5bf832]
/lib/libc.so.6(clone+0x5e)[0x4fef6e]
======= Memory map: ========
00110000-00142000 r-xp 00000000 08:05 1509610    /opt/pcre/lib/libpcre.so.0.0.1
00142000-00143000 rw-p 00031000 08:05 1509610    /opt/pcre/lib/libpcre.so.0.0.1
00143000-0016a000 r-xp 00000000 08:05 163907     /lib/libm-2.5.so
0016a000-0016b000 r--p 00026000 08:05 163907     /lib/libm-2.5.so
0016b000-0016c000 rw-p 00027000 08:05 163907     /lib/libm-2.5.so
0016c000-0018f000 r-xp 00000000 08:05 1508652    /opt/lua/lib/liblua-5.1.3.so
0018f000-00190000 rw-p 00022000 08:05 1508652    /opt/lua/lib/liblua-5.1.3.so
00190000-00191000 r-xp 00000000 08:03 1743490    
/usr/local/apache/modules/mod_auth_passthrough.so
00191000-00192000 rw-p 00000000 08:03 1743490    
/usr/local/apache/modules/mod_auth_passthrough.so
00192000-00193000 r-xp 00000000 08:03 1743501    
/usr/local/apache/modules/mod_bwlimited.so
00193000-00194000 rw-p 00000000 08:03 1743501    
/usr/local/apache/modules/mod_bwlimited.so
00194000-00198000 r-xp 00000000 08:03 1743503    
/usr/local/apache/modules/mod_suphp.so
00198000-00199000 rw-p 00003000 08:03 1743503    
/usr/local/apache/modules/mod_suphp.so
001ef000-001f9000 r-xp 00000000 08:05 165366     /lib/libnss_files-2.5.so
001f9000-001fa000 r--p 00009000 08:05 165366     /lib/libnss_files-2.5.so
001fa000-001fb000 rw-p 0000a000 08:05 165366     /lib/libnss_files-2.5.so
001fb000-00376000 r-xp 00000000 08:03 1736705    
/usr/local/apache/modules/mod_pagespeed.so
00376000-0037f000 rw-p 0017b000 08:03 1736705    
/usr/local/apache/modules/mod_pagespeed.so
0037f000-00380000 rw-p 0037f000 00:00 0 
003a6000-003aa000 r-xp 00000000 08:05 163910     /lib/libnss_dns-2.5.so
003aa000-003ab000 r--p 00003000 08:05 163910     /lib/libnss_dns-2.5.so
003ab000-003ac000 rw-p 00004000 08:05 163910     /lib/libnss_dns-2.5.so
0040e000-00429000 r-xp 00000000 08:05 163903     /lib/ld-2.5.so
00429000-0042a000 r--p 0001a000 08:05 163903     /lib/ld-2.5.so
0042a000-0042b000 rw-p 0001b000 08:05 163903     /lib/ld-2.5.so
0042d000-00580000 r-xp 00000000 08:05 163911     /lib/libc-2.5.so
00580000-00582000 r--p 00152000 08:05 163911     /lib/libc-2.5.so
00582000-00583000 rw-p 00154000 08:05 163911     /lib/libc-2.5.so
00583000-00586000 rw-p 00583000 00:00 0 
00588000-0058b000 r-xp 00000000 08:05 164692     /lib/libdl-2.5.so
0058b000-0058c000 r--p 00002000 08:05 164692     /lib/libdl-2.5.so
0058c000-0058d000 rw-p 00003000 08:05 164692     /lib/libdl-2.5.so
005ab000-005ac000 r-xp 005ab000 00:00 0          [vdso]
005ba000-005cf000 r-xp 00000000 08:05 165413     /lib/libpthread-2.5.so
005cf000-005d0000 r--p 00015000 08:05 165413     /lib/libpthread-2.5.so
005d0000-005d1000 rw-p 00016000 08:05 165413     /lib/libpthread-2.5.so
005d1000-005d3000 rw-p 005d1000 00:00 0 
005d5000-005e7000 r-xp 00000000 08:03 592839     /usr/lib/libz.so.1.2.3
005e7000-005e8000 rw-p 00011000 08:03 592839     /usr/lib/libz.so.1.2.3
005ea000-005f1000 r-xp 00000000 08:05 165416     /lib/librt-2.5.so
005f1000-005f2000 r--p 00007000 08:05 165416     /lib/librt-2.5.so
005f2000-005f3000 rw-p 00008000 08:05 165416     /lib/librt-2.5.so
0063a000-00665000 r-xp 00000000 08:03 1738863    
/usr/local/apache/lib/libapr-1.so.0.4.2
00665000-00666000 rw-p 0002b000 08:03 1738863    
/usr/local/apache/lib/libapr-1.so.0.4.2
00672000-0067b000 r-xp 00000000 08:05 164697     /lib/libcrypt-2.5.so
0067b000-0067c000 r--p 00008000 08:05 164697     /lib/libcrypt-2.5.so
0067c000-0067d000 rw-p 00009000 08:05 164697     /lib/libcrypt-2.5.so
0067d000-006a4000 rw-p 0067d000 00:00 0 
006a6000-006a9000 r-xp 00000000 08:05 165344     /lib/libuuid.so.1.2
006a9000-006aa000 rw-p 00003000 08:05 165344     /lib/libuuid.so.1.2
006ae000-006b9000 r-xp 00000000 08:05 164693     
/lib/libgcc_s-4.1.2-20080825.so.1
006b9000-006ba000 rw-p 0000a000 08:05 164693     
/lib/libgcc_s-4.1.2-20080825.so.1
006f9000-00735000 r-xp 00000000 08:05 1508772    
/opt/openssl/lib/libssl.so.0.9.8
00735000-00739000 rw-p 0003b000 08:05 1508772    
/opt/openssl/lib/libssl.so.0.9.8
00739000-00819000 r-xp 00000000 08:03 592485     /usr/lib/libstdc++.so.6.0.8
00819000-0081d000 r--p 000df000 08:03 592485     /usr/lib/libstdc++.so.6.0.8
0081d000-0081e000 rw-p 000e3000 08:03 592485     /usr/lib/libstdc++.so.6.0.8
0081e000-00824000 rw-p 0081e000 00:00 0 
008a7000-008b7000 r-xp 00000000 08:05 165336     /lib/libresolv-2.5.so
008b7000-008b8000 r--p 0000f000 08:05 165336     /lib/libresolv-2.5.so
008b8000-008b9000 rw-p 00010000 08:05 165336     /lib/libresolv-2.5.so
008b9000-008bb000 rw-p 008b9000 00:00 0 
008bd000-008dc000 r-xp 00000000 08:05 164039     /lib/libexpat.so.0.5.0
008dc000-008de000 rw-p 0001e000 08:05 164039     /lib/libexpat.so.0.5.0
00921000-00a4b000 r-xp 00000000 08:05 1508770    
/opt/openssl/lib/libcrypto.so.0.9.8
00a4b000-00a61000 rw-p 00129000 08:05 1508770    
/opt/openssl/lib/libcrypto.so.0.9.8
00a61000-00a64000 rw-p 00a61000 00:00 0 
00a64000-00b7d000 r-xp 00000000 08:05 1509736    /opt/xml2/lib/libxml2.so.2.7.6
00b7d000-00b82000 rw-p 00118000 08:05 1509736    /opt/xml2/lib/libxml2.so.2.7.6
00b82000-00b83000 rw-p 00b82000 00:00 0 
00c53000-00c74000 r-xp 00000000 08:03 1739155    
/usr/local/apache/lib/libaprutil-1.so.0.3.10
00c74000-00c75000 rw-p 00020000 08:03 1739155    
/usr/local/apache/lib/libaprutil-1.so.0.3.10
00c9c000-00cda000 r-xp 00000000 08:03 1743511    
/usr/local/apache/modules/mod_security2.so
00cda000-00cdc000 rw-p 0003e000 08:03 1743511    
/usr/local/apache/modules/mod_security2.so
00e3a000-00e42000 r-xp 00000000 08:03 1743504    
/usr/local/apache/modules/mod_frontpage.so
00e42000-00e43000 rw-p 00008000 08:03 1743504    
/usr/local/apache/modules/mod_frontpage.so
00e43000-00e44000 ---p 00e43000 00:00 0 
00e44000-01844000 rwxp 00e44000 00:00 0 
01844000-01845000 ---p 01844000 00:00 0 
01845000-02245000 rwxp 01845000 00:00 0 
0269d000-0269e000 ---p 0269d000 00:00 0 
0269e000-0309e000 rwxp 0269e000 00:00 0 
0309e000-0309f000 ---p 0309e000 00:00 0 
0309f000-03a9f000 rwxp 0309f000 00:00 0 
03a9f000-03aa0000 ---p 03a9f000 00:00 0 
03aa0000-044a0000 rwxp 03aa0000 00:00 0 
044a0000-044a1000 ---p 044a0000 00:00 0 
044a1000-04ea1000 rwxp 044a1000 00:00 0 
05407000-05408000 ---p 05407000 00:00 0 
05408000-05e08000 rwxp 05408000 00:00 0 
06457000-06458000 ---p 06457000 00:00 0 
06458000-06e58000 rwxp 06458000 00:00 0 
06e58000-06e59000 ---p 06e58000 00:00 0 
06e59000-07859000 rwxp 06e59000 00:00 0 
08048000-08165000 r-xp 00000000 08:03 1739248    /usr/local/apache/bin/httpd
08165000-08169000 rw-p 0011d000 08:03 1739248    /usr/local/apache/bin/httpd
08169000-0816e000 rw-p 08169000 00:00 0 
0867b000-0867c000 ---p 0867b000 00:00 0 
0867c000-0907c000 rwxp 0867c000 00:00 0 
09650000-0e7ee000 rw-p 09650000 00:00 0          [heap]
abdfe000-abdff000 ---p abdfe000 00:00 0 
abdff000-ac7ff000 rwxp abdff000 00:00 0 
ac7ff000-ac800000 ---p ac7ff000 00:00 0 
ac800000-ad200000 rwxp ac800000 00:00 0 
ad200000-ad23b000 rw-p ad200000 00:00 0 
ad23b000-ad300000 ---p ad23b000 00:00 0 
ad300000-ad3b8000 rw-p ad300000 00:00 0 
ad3b8000-ad400000 ---p ad3b8000 00:00 0 
ad400000-ad44b000 rw-p ad400000 00:00 0 
ad44b000-ad500000 ---p ad44b000 00:00 0 
ad566000-ad567000 ---p ad566000 00:00 0 
ad567000-adf67000 rwxp ad567000 00:00 0 
adf67000-adf68000 ---p adf67000 00:00 0 
adf68000-ae968000 rwxp adf68000 00:00 0 
ae968000-ae969000 ---p ae968000 00:00 0 
ae969000-af369000 rwxp ae969000 00:00 0 
af369000-af36a000 ---p af369000 00:00 0 
af36a000-afd6a000 rwxp af36a000 00:00 0 
afd6a000-afd6b000 ---p afd6a000 00:00 0 
afd6b000-b076b000 rwxp afd6b000 00:00 0 
b076b000-b076c000 ---p b076b000 00:00 0 
b076c000-b116c000 rwxp b076c000 00:00 0 
b116c000-b116d000 ---p b116c000 00:00 0 
b116d000-b1b6d000 rwxp b116d000 00:00 0 
b1b6d000-b1b6e000 ---p b1b6d000 00:00 0 
b1b6e000-b256e000 rwxp b1b6e000 00:00 0 
b256e000-b256f000 ---p b256e000 00:00 0 
b256f000-b2f6f000 rwxp b256f000 00:00 0 
b2f6f000-b2f70000 ---p b2f6f000 00:00 0 
b2f70000-b3970000 rwxp b2f70000 00:00 0 
b3970000-b3971000 ---p b3970000 00:00 0 
b3971000-b4371000 rwxp b3971000 00:00 0 
b4371000-b4372000 ---p b4371000 00:00 0 
b4372000-b4d72000 rwxp b4372000 00:00 0 
b4d72000-b4d73000 ---p b4d72000 00:00 0 
b4d73000-b5773000 rwxp b4d73000 00:00 0 
b5773000-b5774000 ---p b5773000 00:00 0 
b5774000-b6174000 rwxp b5774000 00:00 0 
b6174000-b6175000 ---p b6174000 00:00 0 
b6175000-b6b75000 rwxp b6175000 00:00 0 
b6b75000-b6b76000 ---p b6b75000 00:00 0 
b6b76000-b7576000 rwxp b6b76000 00:00 0 
b7576000-b7577000 ---p b7576000 00:00 0 
b7577000-b7f77000 rwxp b7577000 00:00 0 
b7f77000-b7fb4000 rw-s 00000000 00:09 61335417   /dev/zero (deleted)
b7fb4000-b7fb5000 rw-s 00000000 00:09 23789596   /SYSV01039aa5 (deleted)
b7fb5000-b7fb6000 rw-s 00000000 00:09 23756827   /SYSV01039aa4 (deleted)
b7fb6000-b7fb7000 rw-s 00000000 00:09 23724058   /SYSV01039aa3 (deleted)
b7fb7000-b7fb8000 rw-s 00000000 00:09 23691289   /SYSV01039aa2 (deleted)
b7fb8000-b7fb9000 rw-s 00000000 00:09 23658520   /SYSV01039aa1 (deleted)
b7fb9000-b7fba000 rw-s 00000000 00:09 23625751   /SYSV01039aa0 (deleted)
b7fba000-b7fbb000 rw-s 00000000 00:09 23592982   /SYSV01039a9f (deleted)
b7fbb000-b7fbc000 rw-s 00000000 00:09 23560213   /SYSV01039a9e (deleted)
b7fbc000-b7fbd000 rw-s 00000000 00:09 23527444   /SYSV01039a9d (deleted)
b7fbd000-b7fbe000 rw-s 00000000 00:09 23494675   /SYSV01039a9c (deleted)
b7fbe000-b7fbf000 rw-s 00000000 00:09 23461906   /SYSV01039a9b (deleted)
b7fbf000-b7fc0000 rw-s 00000000 00:09 23429137   /SYSV01039a9a (deleted)
b7fc0000-b7fc1000 rw-s 00000000 00:09 23396368   /SYSV01039a99 (deleted)
b7fc1000-b7fc2000 rw-s 00000000 00:09 23363599   /SYSV01039a44 (deleted)
b7fc2000-b7fc3000 rw-s 00000000 00:09 23330830   /SYSV01039a43 (deleted)
b7fc3000-b7fc7000 rw-p b7fc3000 00:00 0 
b7fc7000-b7fc8000 rw-s 00000000 00:09 23298061   /SYSV010398a8 (deleted)
b7fc8000-b7fc9000 rw-s 00000000 00:09 23265292   /SYSV010398a7 (deleted)
b7fc9000-b7fca000 rw-s 00000000 00:09 23232523   /SYSV010398a5 (deleted)
b7fca000-b7fcb000 rw-s 00000000 00:09 23199754   /SYSV010398a4 (deleted)
b7fcb000-b7fcc000 rw-s 00000000 00:09 23166985   /SYSV0103979a (deleted)
b7fcc000-b7fcd000 rw-s 00000000 00:09 23134216   /SYSV01039797 (deleted)
b7fcd000-b7fce000 rw-s 00000000 00:09 23101447   /SYSV01038cdf (deleted)
b7fce000-b7fcf000 rw-s 00000000 00:09 23068678   /SYSV01038cd4 (deleted)
b7fcf000-b7fd0000 rw-s 00000000 00:09 23035909   /SYSV010388a2 (deleted)
b7fd0000-b7fd1000 rw-s 00000000 00:09 23003140   /SYSV010388a1 (deleted)
b7fd1000-b7fd2000 rw-s 00000000 00:09 22970369   /SYSV0103887e (deleted)
b7fd2000-b7fd3000 rw-p b7fd2000 00:00 0 
bf94f000-bf963000 rwxp bffea000 00:00 0          [stack]
bf963000-bf964000 rw-p bfffe000 00:00 0 
[Tue Nov 09 20:09:09 2010] [error] log_message_handler: 
[1109/200909:ERROR:net/instaweb/apache/apache_rewrite_driver_factory.cc(154)] 
Remove rewrite driver from the active list.
[Tue Nov 09 20:09:09 2010] [notice] child pid 11851 exit signal Segmentation 
fault (11)

Original issue reported on code.google.com by cubecity...@gmail.com on 9 Nov 2010 at 7:58

[GoogleCodeExporter]

Hi,

We have not tested yet with the Event MPM or Threaded MPM.  We would love to 
support those environments but it's not done yet.

For now, we only have mod_pagespeed working using the Apache pre-fork MPM.

Inspecting the code it's not immediately obvious why this would fail for Event 
MPM but it's certainly not something we've tested yet.

Original comment by jmara...@google.com on 9 Nov 2010 at 9:24

• Changed title: Apache children segfaulting using Event MPM

[GoogleCodeExporter]

I inspected one more time and I have a theory about why this would break in 
threaded or event MPMs.  One mutex coming up :)

Original comment by jmara...@google.com on 9 Nov 2010 at 9:36

[GoogleCodeExporter]

I have checked in a change that may improve the behavior on Threaded and Event 
MPMs, but there no guarantees as I am not set up to test this.

This is available only as a source-code build from trunk, e.g.

gclient config http://modpagespeed.googlecode.com/svn/trunk/src

Original comment by jmara...@google.com on 9 Nov 2010 at 11:32

[GoogleCodeExporter]

In 0.9.8.1-r215 we found & fixed a memory corruption issue.  However we still 
have not tested the event or threaded MPMs.  I will add support for those as 
separate enhancments.

Original comment by jmara...@google.com on 17 Nov 2010 at 4:47

• Changed state: Fixed