New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apache2 coredump after enabling mod_pagespeed's extend_cache on centOS. #610

Closed
GoogleCodeExporter opened this Issue Apr 6, 2015 · 9 comments

Comments

Projects
None yet
1 participant
@GoogleCodeExporter
Copy link

GoogleCodeExporter commented Apr 6, 2015

This is a cross post from my question on serverfault: 
http://serverfault.com/questions/461448/apache2-coredump-after-enabling-mod-page
speed-on-centos

I'm trying to get the following apache2.2 modules to work together 
mod_pagespeed, mod_spdy and WebSphere Webserver plugin on CentOS.

Once mod_pagespeed changes the path of a file the browser can't resolve that 
file anymore. Enable or disable mod_spdy doesn't change any behaviour. 
mod_pagespeed with extend_cache works (no coredumps) if I disable WebSphere 
plugin. 


Response from Eric Covener:
it looks like a mod_pagespeed bug (null pointer dereference) hen r->filename is 
not set by the handler. mod_proxy sets it to proxy:$originaluri, the WAS Plugin 
doesn't set it it at all.

I have a coredump but I don't really know how to read this:
gdb /usr/sbin/httpd core.881
(gdb) bt full
#0 __strrchr_sse42 () at ../sysdeps/x86_64/multiarch/strrchr.S:134
No locals.
#1 0x00007ff129b02404 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
No symbol table info available.
#2 0x00007ff1348fb8b8 in ap_run_map_to_storage (r=0x7ff1362f0698) at 
/usr/src/debug/httpd-2.2.15/server/request.c:69
pHook = <value optimized out>
n = <value optimized out>
rv = <value optimized out>
#3 0x00007ff1348fd9c8 in ap_process_request_internal (r=0x7ff1362f0698) at 
/usr/src/debug/httpd-2.2.15/server/request.c:150
file_req = 0
access_status = <value optimized out>
#4 0x00007ff13490fa20 in ap_process_request (r=0x7ff1362f0698) at 
/usr/src/debug/httpd-2.2.15/modules/http/http_request.c:280
access_status = <value optimized out>
#5 0x00007ff13490c8f8 in ap_process_http_connection (c=0x7ff1362f46b8) at 
/usr/src/debug/httpd-2.2.15/modules/http/http_core.c:190
r = 0x7ff1362f0698
csd = 0x0
#6 0x00007ff134908608 in ap_run_process_connection (c=0x7ff1362f46b8) at 
/usr/src/debug/httpd-2.2.15/server/connection.c:43
pHook = <value optimized out>
n = <value optimized out>
rv = <value optimized out>
#7 0x00007ff1291272fb in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#8 0x00007ff129152ccd in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#9 0x00007ff129152ccd in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#10 0x00007ff12914f1fb in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#11 0x00007ff1291328f1 in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#12 0x00007ff1331ab851 in start_thread (arg=0x7ff12735d700) at 
pthread_create.c:301
__res = <value optimized out>
pd = 0x7ff12735d700
now = <value optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140673721685760, 
-6615953477917025934, 140734441252576, 140673721686464, 4, 7, 
6614986543813548402, 6614960367733758322},
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
cleanup = 0x0, canceltype = 0}}}
not_first_call = <value optimized out>
pagesize_m1 = <value optimized out>
sp = <value optimized out>
freesize = <value optimized out>
#13 0x00007ff132ef911d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.

Original issue reported on code.google.com by k...@allnice.be on 23 Jan 2013 at 11:21

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

I think this might be fixed already in trunk.  Would it be possible for you to 
build from trunk & try it out?

Specifically, special handling was added for reverse-proxies to bypass this 
routine.  It wasn't specifically because r->filename was NULL; we were not 
seeing a crash, but were instead having issues with 403s.  This was Issue 582.

Original comment by jmara...@google.com on 23 Jan 2013 at 12:08

  • Changed state: RequestClarification
@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Building from trunk: 
https://developers.google.com/speed/docs/mod_pagespeed/build_from_source

Follow the commands for the 'bleeding edge' version.

Original comment by jmara...@google.com on 23 Jan 2013 at 12:14

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

I think I've build from bleeding edge source and this is what I got:

In response:
  x-mod-pagespeed:1.1.0.0-2410
  x-mod-spdy:0.9.3.3-386

Now I have multiple coredumps per page refresh:
(gdb) bt full
#0  __strrchr_sse42 () at ../sysdeps/x86_64/multiarch/strrchr.S:134
No locals.
#1  0x00007f9c2d08756c in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
No symbol table info available.
#2  0x00007f9c380708b8 in ap_run_map_to_storage (r=0x7f9c39c935d8) at 
/usr/src/debug/httpd-2.2.15/server/request.c:69
        pHook = <value optimized out>
        n = <value optimized out>
        rv = <value optimized out>
#3  0x00007f9c380729c8 in ap_process_request_internal (r=0x7f9c39c935d8) at 
/usr/src/debug/httpd-2.2.15/server/request.c:150
        file_req = 0
        access_status = <value optimized out>
#4  0x00007f9c38084a20 in ap_process_request (r=0x7f9c39c935d8) at 
/usr/src/debug/httpd-2.2.15/modules/http/http_request.c:280
        access_status = <value optimized out>
#5  0x00007f9c380818f8 in ap_process_http_connection (c=0x7f9c39c8f5b8) at 
/usr/src/debug/httpd-2.2.15/modules/http/http_core.c:190
        r = 0x7f9c39c935d8
        csd = 0x0
#6  0x00007f9c3807d608 in ap_run_process_connection (c=0x7f9c39c8f5b8) at 
/usr/src/debug/httpd-2.2.15/server/connection.c:43
        pHook = <value optimized out>
        n = <value optimized out>
        rv = <value optimized out>
#7  0x00007f9c2c68d2fb in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#8  0x00007f9c2c6b8ccd in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#9  0x00007f9c2c6b8ccd in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#10 0x00007f9c2c6b51fb in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#11 0x00007f9c2c6988f1 in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#12 0x00007f9c36920851 in start_thread (arg=0x7f9c2aaad700) at 
pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7f9c2aaad700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140308707464960, -7206163316611682514, 140736396973904, 140308707465664, 4, 7, 7261709088800173870, 7261770178195156782},
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#13 0x00007f9c3666e11d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.

Original comment by k...@allnice.be on 23 Jan 2013 at 2:32

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

When you did the 'make' did you do 'make BUILDTYPE=Release' ?

The stack trace still shows no debug so you're using a Release build, but to 
get line numbers you need a Debug build: make BUILDTYPE=Debug

Original comment by matterb...@google.com on 23 Jan 2013 at 2:36

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

I'm not sure I did everything correct but here is my new coredump:

(gdb) bt full
#0  __strrchr_sse42 () at ../sysdeps/x86_64/multiarch/strrchr.S:134
No locals.
#1  0x00007f0597e71f98 in net_instaweb::instaweb_map_to_storage 
(request=0x7f05a4444a08) at net/instaweb/apache/instaweb_handler.cc:993
        filename_starting_at_last_slash = 0x7f05a4472bf0 "\235\t0\243\005\177"
        ret = -1
#2  0x00007f05a349a8b8 in ap_run_map_to_storage (r=0x7f05a4444a08) at 
/usr/src/debug/httpd-2.2.15/server/request.c:69
        pHook = <value optimized out>
        n = <value optimized out>
        rv = <value optimized out>
#3  0x00007f05a349c9c8 in ap_process_request_internal (r=0x7f05a4444a08) at 
/usr/src/debug/httpd-2.2.15/server/request.c:150
        file_req = 0
        access_status = <value optimized out>
#4  0x00007f05a34aea20 in ap_process_request (r=0x7f05a4444a08) at 
/usr/src/debug/httpd-2.2.15/modules/http/http_request.c:280
        access_status = <value optimized out>
#5  0x00007f05a34ab8f8 in ap_process_http_connection (c=0x7f05a44409e8) at 
/usr/src/debug/httpd-2.2.15/modules/http/http_core.c:190
        r = 0x7f05a4444a08
        csd = 0x0
#6  0x00007f05a34a7608 in ap_run_process_connection (c=0x7f05a44409e8) at 
/usr/src/debug/httpd-2.2.15/server/connection.c:43
        pHook = <value optimized out>
        n = <value optimized out>
        rv = <value optimized out>
#7  0x00007f05971c02fb in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#8  0x00007f05971ebccd in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#9  0x00007f05971ebccd in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#10 0x00007f05971e81fb in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#11 0x00007f05971cb8f1 in ?? () from /usr/lib64/httpd/modules/mod_spdy.so
No symbol table info available.
#12 0x00007f05a1d4a851 in start_thread (arg=0x7f0594bdf700) at 
pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7f0594bdf700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139661947041536, 3703545060135028190, 140735242467712, 139661947042240, 4, 7, -3643821220001987106, -3643727428746983970},
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#13 0x00007f05a1a9811d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.

Original comment by k...@allnice.be on 23 Jan 2013 at 3:45

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

OK I think we just need to add the null-check as well.  Evidently WebSphere or 
whatever sets r->filename=NULL but does not set r->proxyreq==PROXYREQ_REVERSE, 
so we just need to change line 960 to:

  if ((request->proxyreq == PROXYREQ_REVERSE) || (request->filename == NULL)) {
...

Would that be easy for you to try?

Original comment by jmara...@google.com on 23 Jan 2013 at 3:48

  • Changed state: Accepted
@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

[deleted comment]
@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Seem to be working now \o/

Original comment by k...@allnice.be on 23 Jan 2013 at 4:03

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Fixed as of r2448.

Original comment by matterb...@google.com on 29 Jan 2013 at 7:51

  • Changed state: Fixed
  • Added labels: Milestone-v25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment