Old worker processes segfault on exit during reload

[dotsam]

Running pagespeed 1.10.34.5 with nginx 1.8.1 on CentOS 7, I'm getting a segfault on reload (SIGHUP). Backtrace points to a problem in ps_exit_child_process, which looks to have been changed in d7f1c0d

Running the same exact config with 1.9.32.10 and there is no issue. All debug info below.

uname -a
Linux 3.10.0-327.4.5.el7.x86_64 #1 SMP Mon Jan 25 22:07:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

/etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

Nginx version

nginx version: nginx/1.8.1
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --with-debug --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nobody --group=nobody --add-module=naxsi-0.54/naxsi_src --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-file-aio --with-threads --with-ipv6 --with-http_spdy_module --add-module=/usr/local/rvm/gems/ruby-2.3.0/gems/passenger-5.0.24/src/nginx_module --add-module=ngx_pagespeed-release-1.10.33.5-beta --add-module=ngx_cache_purge-2.3 --with-cc-opt='-pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'

Debug log
https://gist.github.com/dotsam/d0474571774ceb6ae46a

Syslog

Feb 16 20:41:42  kernel: nginx[8919]: segfault at 0 ip 000000000052b4b2 sp 00007fffec8e6ba0 error 4
Feb 16 20:41:42  kernel: nginx[8984]: segfault at 0 ip 000000000052b4b2 sp 00007fffec8e6ba0 error 4
Feb 16 20:41:42  kernel:  in nginx[400000+fef000]
Feb 16 20:41:42  kernel: 
Feb 16 20:41:42  kernel: nginx[9022]: segfault at 0 ip 000000000052b4b2 sp 00007fffec8e6ba0 error 4
Feb 16 20:41:42  kernel:  in nginx[400000+fef000]
Feb 16 20:41:42  kernel: 
Feb 16 20:41:42  kernel: nginx[8921]: segfault at 0 ip 000000000052b4b2 sp 00007fffec8e6ba0 error 4
Feb 16 20:41:42  kernel:  in nginx[400000+fef000]
Feb 16 20:41:42  kernel: 
Feb 16 20:41:42  kernel: nginx[8918]: segfault at 0 ip 000000000052b4b2 sp 00007fffec8e6ba0 error 4
Feb 16 20:41:42  kernel:  in nginx[400000+fef000]
Feb 16 20:41:42  kernel: 
Feb 16 20:41:42  kernel:  in nginx[400000+fef000]
Feb 16 20:41:42  kernel: 
Feb 16 20:41:42  kernel: nginx[8946]: segfault at 0 ip 000000000052b4b2 sp 00007fffec8e6ba0 error 4 in nginx[400000+fef000]
Feb 16 20:41:42  kernel: nginx[8922]: segfault at 0 ip 000000000052b4b2 sp 00007fffec8e6ba0 error 4 in nginx[400000+fef000]
Feb 16 20:41:42  kernel: nginx[8920]: segfault at 0 ip 000000000052b4b2 sp 00007fffec8e6ba0 error 4 in nginx[400000+fef000]
Feb 16 20:41:46  kernel: nginx[9194]: segfault at 0 ip 000000000052b4b2 sp 00007fffec8e6ba0 error 4 in nginx[400000+fef000]
Feb 16 20:41:46  kernel: nginx[9422]: segfault at 0 ip 000000000052b4b2 sp 00007fffec8e6ba0 error 4 in nginx[400000+fef000]

Backtrace

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/nginx...done.
[New LWP 8918]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `nginx: worker process is shutting down  '.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000052b4b2 in net_instaweb::(anonymous namespace)::ps_exit_child_process (cycle=0x1afacb0)
    at ngx_pagespeed-release-1.10.33.5-beta/src/ngx_pagespeed.cc:3071
3071      cfg_m->driver_factory->ShutDown();
Missing separate debuginfos, use: debuginfo-install nginx-nDeploy-1.8.1-4.el7.x86_64
(gdb) backtrace full
#0  0x000000000052b4b2 in net_instaweb::(anonymous namespace)::ps_exit_child_process (cycle=0x1afacb0)
    at ngx_pagespeed-release-1.10.33.5-beta/src/ngx_pagespeed.cc:3071
        cfg_m = 0x1afec40
#1  0x000000000044be9c in ngx_worker_process_exit (cycle=0x1afacb0) at src/os/unix/ngx_process_cycle.c:980
        i = 54
        c = 0x0
#2  0x000000000044b4c1 in ngx_worker_process_cycle (cycle=0x1afacb0, data=0x0) at src/os/unix/ngx_process_cycle.c:761
        worker = 0
        i = 1024
        c = 0x1bad820
#3  0x0000000000447ccb in ngx_spawn_process (cycle=0x1afacb0, proc=0x44b325 <ngx_worker_process_cycle>, data=0x0, name=0xdfc22b "worker process", respawn=-3)
    at src/os/unix/ngx_process.c:198
        on = 1
        pid = 0
        s = 0
#4  0x000000000044a27d in ngx_start_worker_processes (cycle=0x1afacb0, n=8, type=-3) at src/os/unix/ngx_process_cycle.c:357
        i = 0
        ch = {command = 1, pid = 0, slot = 0, fd = 0}
#5  0x0000000000449890 in ngx_master_process_cycle (cycle=0x1afacb0) at src/os/unix/ngx_process_cycle.c:129
        title = 0x1bab9cc "master process /usr/sbin/nginx -c /etc/nginx/nginx.conf"
        p = 0x1baba03 ""
        size = 56
        i = 3
        n = 0
        sigio = 0
        set = {__val = {0 <repeats 16 times>}}
        itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}
        live = 0
        delay = 0
        ls = 0x0
        ccf = 0x1afbdb8
#6  0x0000000000413665 in main (argc=3, argv=0x7fffec8e7128) at src/core/nginx.c:419
        i = 66
        log = 0x165fa40 <ngx_log>
        cycle = 0x1afacb0
        init_cycle = {conf_ctx = 0x0, pool = 0x1afa6e0, log = 0x165fa40 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0, 
            handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0, 
          free_connection_n = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, 
          paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, 
            nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, 
          files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21, 
---Type <return> to continue, or q <return> to quit---
            data = 0x7fffec8e7f8a "ss is shutting down"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, 
            data = 0x7fffec8e7f8a "ss is shutting down"}, prefix = {len = 11, data = 0xdf7e68 "/etc/nginx/"}, lock_file = {len = 0, data = 0x0}, hostname = {
            len = 0, data = 0x0}}
        ccf = 0x1afbdb8

[oschaaf]

@dotsam Are you able to reproduce this consistently? Could you post or PM your (anonymized) nginx.conf?

[oschaaf]

Reproduced, this happens when ngx_pagespeed is completely turned off, at which point cfg_m->driver_factory is NULL.
Thanks for the report, I will send a pull request shortly to fix this.