Crash on very long urls

[praveen262k]

We enabled pagespeed module 2 weeks back. We have been monitoring site since then.

Last 2 days, we see below logs and server goes not responding. Any help would be appreciated! As of now we have pulled out the module from production.

GDB Backtrace log:

(gdb) bt
#0  0x00007f54cfd95625 in raise () from /lib64/libc.so.6
#1  0x00007f54cfd96e05 in abort () from /lib64/libc.so.6
#2  0x00007f54cad1fa5d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3  0x00007f54cad1dbe6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00007f54cad1dc13 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x00007f54cad1dd0e in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x00007f54cacc2837 in std::__throw_logic_error(char const*) () from /usr/lib64/libstdc++.so.6
#7  0x00007f54cacfde59 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007f54cacfdf33 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) () from /usr/lib64/libstdc++.so.6
#9  0x00007f54c9fce073 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#10 0x00007f54c9effb9a in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#11 0x00007f54c9f01e34 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#12 0x00007f54c9f0b3a4 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#13 0x00007f54d1860a58 in ap_send_error_response ()
#14 0x00007f54d1848506 in ap_read_request ()
#15 0x00007f54d18600e0 in ?? ()
#16 0x00007f54d185c148 in ap_run_process_connection ()
#17 0x00007f54d1869052 in ?? ()
#18 0x00007f54d00fe9d1 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f54cfe4b8fd in clone () from /lib64/libc.so.6

Apache log:

$ grep "child pid" error.20160118000000.log
[Sun Jan 17 20:52:31 2016] [notice] child pid 12587 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 08:51:39 2016] [notice] child pid 14373 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 09:36:22 2016] [notice] child pid 473 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 09:40:47 2016] [notice] child pid 24587 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:32 2016] [notice] child pid 29667 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:32 2016] [notice] child pid 8104 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:32 2016] [notice] child pid 8695 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:44 2016] [notice] child pid 18148 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:46 2016] [notice] child pid 22018 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:49 2016] [notice] child pid 22962 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:49 2016] [notice] child pid 23747 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:49 2016] [notice] child pid 23748 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:52 2016] [notice] child pid 25136 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:54 2016] [notice] child pid 25719 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:54 2016] [notice] child pid 25720 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:54 2016] [notice] child pid 27435 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:54 2016] [notice] child pid 27436 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:54 2016] [notice] child pid 27443 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:53:04 2016] [notice] child pid 32360 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:53:14 2016] [notice] child pid 9726 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:53:14 2016] [notice] child pid 9727 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 10:53:22 2016] [notice] child pid 14263 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:50 2016] [notice] child pid 20134 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:50 2016] [notice] child pid 21619 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:50 2016] [notice] child pid 26322 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:51 2016] [notice] child pid 3596 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:51 2016] [notice] child pid 28482 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:53 2016] [notice] child pid 6108 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 12:58:02 2016] [notice] child pid 6925 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 12:58:07 2016] [notice] child pid 8408 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 12:58:13 2016] [notice] child pid 7685 exit signal Aborted (6), possible coredump in /etc/httpd
[Mon Jan 18 12:58:14 2016] [notice] child pid 9603 exit signal Aborted (6), possible coredump in /etc/httpd

[crowell]

What version/os/64 or 32 bits?
On Jan 19, 2016 5:51 PM, "praveen262k" notifications@github.com wrote:

We enabled pagespeed module 2 weeks back. We have been monitoring site
since then.

Last 2 days, we see below logs and server goes not responding. Any help
would be appreciated! As of now we have pulled out the module from
production.
GDB Backtrace log:

(gdb) bt
#0 0x00007f54cfd95625 in raise () from /lib64/libc.so.6
#1 #1
0x00007f54cfd96e05 in abort () from /lib64/libc.so.6
#2 #2
0x00007f54cad1fa5d in __gnu_cxx::__verbose_terminate_handler() () from
/usr/lib64/libstdc++.so.6
#3 #3
0x00007f54cad1dbe6 in ?? () from /usr/lib64/libstdc++.so.6
#4 #4
0x00007f54cad1dc13 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5 #5
0x00007f54cad1dd0e in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6 #6
0x00007f54cacc2837 in std::__throw_logic_error(char const

) () from /usr/lib64/libstdc++.so.6 #7
#7 0x00007f54cacfde59 in
?? () from /usr/lib64/libstdc++.so.6 #8
#8 0x00007f54cacfdf33 in
std::basic_string, std::allocator >::basic_string(char const,
std::allocator const&) () from /usr/lib64/libstdc++.so.6
#9 #9
0x00007f54c9fce073 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#10 #10
0x00007f54c9effb9a in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#11 #11
0x00007f54c9f01e34 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#12 #12
0x00007f54c9f0b3a4 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#13 #13
0x00007f54d1860a58 in ap_send_error_response ()
#14 #14
0x00007f54d1848506 in ap_read_request ()
#15 #15
0x00007f54d18600e0 in ?? ()
#16 #16
0x00007f54d185c148 in ap_run_process_connection ()
#17 #17
0x00007f54d1869052 in ?? ()
#18 #18
0x00007f54d00fe9d1 in start_thread () from /lib64/libpthread.so.0
#19 #19
0x00007f54cfe4b8fd in clone () from /lib64/libc.so.6

Apache log:
$ grep "child pid" error.20160118000000.log
[Sun Jan 17 20:52:31 2016] [notice] child pid 12587 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 08:51:39 2016] [notice] child pid 14373 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 09:36:22 2016] [notice] child pid 473 exit signal Aborted (6),
possible coredump in /etc/httpd
[Mon Jan 18 09:40:47 2016] [notice] child pid 24587 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:32 2016] [notice] child pid 29667 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:32 2016] [notice] child pid 8104 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:32 2016] [notice] child pid 8695 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:44 2016] [notice] child pid 18148 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:46 2016] [notice] child pid 22018 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:49 2016] [notice] child pid 22962 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:49 2016] [notice] child pid 23747 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:49 2016] [notice] child pid 23748 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:52 2016] [notice] child pid 25136 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:54 2016] [notice] child pid 25719 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:54 2016] [notice] child pid 25720 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:54 2016] [notice] child pid 27435 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:54 2016] [notice] child pid 27436 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:52:54 2016] [notice] child pid 27443 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:53:04 2016] [notice] child pid 32360 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:53:14 2016] [notice] child pid 9726 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:53:14 2016] [notice] child pid 9727 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 10:53:22 2016] [notice] child pid 14263 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:50 2016] [notice] child pid 20134 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:50 2016] [notice] child pid 21619 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:50 2016] [notice] child pid 26322 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:51 2016] [notice] child pid 3596 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:51 2016] [notice] child pid 28482 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 12:57:53 2016] [notice] child pid 6108 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 12:58:02 2016] [notice] child pid 6925 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 12:58:07 2016] [notice] child pid 8408 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 12:58:13 2016] [notice] child pid 7685 exit signal Aborted
(6), possible coredump in /etc/httpd
[Mon Jan 18 12:58:14 2016] [notice] child pid 9603 exit signal Aborted
(6), possible coredump in /etc/httpd

—
Reply to this email directly or view it on GitHub
#1248.

[praveen262k]

Apache version: 2.2
OS: Centos Red Hat Linux 64 bit

Thanks!

[crowell]

Also what version of mod_pagespeed?
On Jan 19, 2016 5:59 PM, "praveen262k" notifications@github.com wrote:

Apache version: 2.2
OS: Centos Red Hat Linux 64 bit

Thanks!

—
Reply to this email directly or view it on GitHub
#1248 (comment)
.

[praveen262k]

ModSpeed version: Version: 1.9.32.11
Thanks!

[praveen262k]

I saw similar issue reported #334 (looks pretty old ticket) - and was mentioned that it was resolved in Release 0.10.22.4 (https://developers.google.com/speed/pagespeed/module/release_notes#release_1.9.32.11-stable)

[praveen262k]

To add a few more information:

1. We have enabled rewrite css, rewrite js and collapse whitespace are only enabled
2. ModePagespeedRewriteLevel PassThrough

[praveen262k]

A few more logs details we pulled from apache log:

Tue Jan 19 04:30:28 2016] [error] [mod_pagespeed 1.9.32.11-7550 @9909] .....
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_S_construct NULL not valid
[Tue Jan 19 04:30:29 2016] [notice] child pid 9909 exit signal Aborted (6), possible coredump in /etc/httpd

[crowell]

great, thanks. I'll symbolize the backtrace tomorrow morning and hopefully
that'll make pinpointing the problem easy.
On Jan 19, 2016 8:13 PM, "praveen262k" notifications@github.com wrote:

A few more logs details we pulled from apache log:

Tue Jan 19 04:30:28 2016] [error] [mod_pagespeed 1.9.32.11-7550 @9909]
.....
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_S_construct NULL not valid
[Tue Jan 19 04:30:29 2016] [notice] child pid 9909 exit signal Aborted
(6), possible coredump in /etc/httpd

—
Reply to this email directly or view it on GitHub
#1248 (comment)
.

[praveen262k]

A few more info:

We noticed that somehow this error is related to the crash (most of the time):
[Tue Jan 19 12:06:16 2016] [error] [client XXX.X.XXX.XX] request failed: URI too long (longer than 8190)

[crowell]

if you still have the process in gdb, could you do info proc mappings?

[praveen262k]

Unfortunately it says proc folder not found

However out team gave some more insight from the coredump and below is details

(gdb) bt
#0  0x00007f54cfd95625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007f54cfd96e05 in abort () at abort.c:92
#2  0x00007f54cad1fa5d in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:93
#3  0x00007f54cad1dbe6 in __cxxabiv1::__terminate (handler=<value optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:38
#4  0x00007f54cad1dc13 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#5  0x00007f54cad1dd0e in __cxxabiv1::__cxa_throw (obj=0x7f52a4010de0, tinfo=<value optimized out>, dest=<value optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:83
#6  0x00007f54cacc2837 in std::__throw_logic_error (__s=<value optimized out>) at ../../../../libstdc++-v3/src/functexcept.cc:60
#7  0x00007f54cacfde59 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_S_construct<char const*> (__beg=0x0, __end=<value optimized out>,
    __a=<value optimized out>) at /usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:134
#8  0x00007f54cacfdf33 in _S_construct_aux<char const*> (this=0x7f54a0bcc580, __s=0x0, __a=...)
    at /usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.h:1543
#9  _S_construct<char const*> (this=0x7f54a0bcc580, __s=0x0, __a=...)
    at /usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.h:1559
#10 std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7f54a0bcc580, __s=0x0, __a=...)
    at /usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:216
#11 0x00007f54c9fce073 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#12 0x00007f54c9effb9a in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#13 0x00007f54c9f01e34 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#14 0x00007f54c9f0b3a4 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#15 0x00007f54d1860a58 in ap_send_error_response (r=0x7f52a8097338, recursive_error=<value optimized out>) at /usr/src/debug/httpd-2.2.15/modules/http/http_protocol.c:1288
#16 0x00007f54d1848506 in ap_read_request (conn=0x7f538c05d3c0) at /usr/src/debug/httpd-2.2.15/server/protocol.c:944
#17 0x00007f54d18600e0 in ap_process_http_connection (c=0x7f538c05d3c0) at /usr/src/debug/httpd-2.2.15/modules/http/http_core.c:183
#18 0x00007f54d185c148 in ap_run_process_connection (c=0x7f538c05d3c0) at /usr/src/debug/httpd-2.2.15/server/connection.c:43
#19 0x00007f54d1869052 in process_socket (thd=0x7f54d2d8a340, dummy=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:544
#20 worker_thread (thd=0x7f54d2d8a340, dummy=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:894
#21 0x00007f54d00fe9d1 in start_thread (arg=0x7f54a0bcd700) at pthread_create.c:301
#22 0x00007f54cfe4b8fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

[crowell]

Can you do info sharedlibrary ?
On Jan 20, 2016 11:51 AM, "praveen262k" notifications@github.com wrote:

Unfortunately it says proc folder not found

However out team gave some more insight from the coredump and below is
details

(gdb) bt
#0 0x00007f54cfd95625 in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 #1
0x00007f54cfd96e05 in abort () at abort.c:92
#2 #2
0x00007f54cad1fa5d in __gnu_cxx::__verbose_terminate_handler () at
../../../../libstdc++-v3/libsupc++/vterminate.cc:93
#3 #3
0x00007f54cad1dbe6 in __cxxabiv1::__terminate (handler=) at
../../../../libstdc++-v3/libsupc++/eh_terminate.cc:38
#4 #4
0x00007f54cad1dc13 in std::terminate () at
../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#5 #5
0x00007f54cad1dd0e in __cxxabiv1::__cxa_throw (obj=0x7f52a4010de0, tinfo=,
dest=) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:83
#6 #6
0x00007f54cacc2837 in std::__throw_logic_error (__s=) at
../../../../libstdc++-v3/src/functexcept.cc:60
#7 #7
0x00007f54cacfde59 in std::basic_string, std::allocator >::_S_construct
(__beg=0x0, __end=,
__a=) at
/usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:134
#8 #8
0x00007f54cacfdf33 in _S_construct_aux (this=0x7f54a0bcc580, __s=0x0,
__a=...)
at
/usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.h:1543
#9 #9 _S_construct
(this=0x7f54a0bcc580, __s=0x0, __a=...)
at
/usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.h:1559
#10 #10
std::basic_string, std::allocator >::basic_string (this=0x7f54a0bcc580,
__s=0x0, __a=...)
at
/usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:216
#11 #11
0x00007f54c9fce073 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#12 #12
0x00007f54c9effb9a in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#13 #13
0x00007f54c9f01e34 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#14 #14
0x00007f54c9f0b3a4 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#15 #15
0x00007f54d1860a58 in ap_send_error_response (r=0x7f52a8097338,
recursive_error=) at
/usr/src/debug/httpd-2.2.15/modules/http/http_protocol.c:1288
#16 #16
0x00007f54d1848506 in ap_read_request (conn=0x7f538c05d3c0) at
/usr/src/debug/httpd-2.2.15/server/protocol.c:944
#17 #17
0x00007f54d18600e0 in ap_process_http_connection (c=0x7f538c05d3c0) at
/usr/src/debug/httpd-2.2.15/modules/http/http_core.c:183
#18 #18
0x00007f54d185c148 in ap_run_process_connection (c=0x7f538c05d3c0) at
/usr/src/debug/httpd-2.2.15/server/connection.c:43
#19 #19
0x00007f54d1869052 in process_socket (thd=0x7f54d2d8a340, dummy=) at
/usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:544
#20 #20 worker_thread
(thd=0x7f54d2d8a340, dummy=) at
/usr/src/debug/httpd-2.2.15/server/mpm/worker/worker.c:894
#21 #21
0x00007f54d00fe9d1 in start_thread (arg=0x7f54a0bcd700) at
pthread_create.c:301
#22 #22
0x00007f54cfe4b8fd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115

—
Reply to this email directly or view it on GitHub
#1248 (comment)
.

[praveen262k]

Please find the attached image - I could not format is well here

[praveen262k]

Some more data!

URI too long -> No of time we see this error log
std::logic_error -> Count of this error messages
child Process -> No of times Child process terminate

We could not find yet why Apache is reporting long URL as we don't see that in our apache access log yet!

[crowell]

is the backtrace from the same process as the info sharedlibrary, the bt doesn't seem to align with anything reasonable here.