crash when database connection fails in presence of fallback

[poeml]

                                                                                            [          ]

Issue migrated (2015-06-05) from old issue tracker http://mirrorbrain.org/issues/issue84

Title    crash when database connection fails in presence of fallback
              mirrors
 Priority   bug                       Status            resolved
Superseder                           Nosy List          poeml
Assigned To poeml                    Keywords

msg323 (view) Author: poeml Date: 2011-02-23.20:50:32

I saw a crash that seems to occur when the database connection fails (for any reason) and
fallback mirrors are configured:

(gdb) thread apply all bt
....

(gdb) thread 24
[Switching to thread 24 (Thread 29704)]#0 0xffffe422 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe422 in __kernel_vsyscall ()
#1 0xb738ccb6 in kill () from /lib/libc.so.6
#2 0xb7795e06 in sig_coredump (sig=11) at mpm_common.c:1231
#3
#4 0xb70f594c in mb_handler (r=0xb798f540) at mod_mirrorbrain.c:1943
#5 0xb778c03d in ap_run_handler (r=0xb798f540) at config.c:158
#6 0xb778fbb4 in ap_invoke_handler (r=0xb798f540) at config.c:376
#7 0xb779c641 in ap_process_request (r=0xb798f540) at http_request.c:282
#8 0xb77990d8 in ap_process_http_connection (c=0xb798b6e0) at http_core.c:190
#9 0xb779449d in ap_run_process_connection (c=0xb798b6e0) at connection.c:43
#10 0xb77a2c2c in worker_thread (thd=0xb77fd538, dummy=0xb78235d0) at worker.c:544
#11 0xb7542b56 in ?? () from /usr/lib/libapr-1.so.0
#12 0xb74ca1b5 in start_thread () from /lib/libpthread.so.0
#13 0xb743362e in clone () from /lib/libc.so.6

(gdb) up
...

(gdb) l
1938 return HTTP_NOT_FOUND;
1939 }
1940 }
1941
1942
1943 statement = apr_hash_get(dbd->prepared, scfg->query_label, APR_HASH_KEY_STRING);
1944
1945 if (statement == NULL) {
1946 ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
1947 "[mod_mirrorbrain] Could not get prepared statement labelled
'%s'",

(gdb) i lo
cfg = (mb_dir_conf ) 0xb7995950
ptr =
uri =
filename = 0xb79964d9 "stable/3.3.0/OOo_3.3.0_MacOS_x86_install_en-US.dmg"
basename = 0xb79964e6 "OOo_3.3.0_MacOS_x86_install_en-US.dmg"
mirror_base = 0xb7996480 "/srv/ooo"
realfile =
yum = (yumdir_t *) 0x0
clientip = 0xb798bb60 "128.232.241.11"
clientaddr =
query_country = 0x0
query_asn = 0x0
fakefile = 0 '\0'
only_hash = 0 '\0'
rep = 0
rep_ext = 0x0
meta_negotiated = 0 '\0'
continent_code = 0xb70b92c7 "EU"
country_code = 0xb70b8bc7 "GB"
country_name = 0xb70b8078 "United Kingdom"
slat =
slng =
lat = 30.4797993
lng = -87.865799
state_id =
state_name =
as = 0xb71003dd "--"
prefix = 0xb71003dd "--"
i = -1215364960
mirror_cnt =
len = 20
nr = 3056701544
new =
mirror =
mirrorp =
chosen =
hashbag = (hashbag_t *) 0x0
rv =
res = (apr_dbd_results_t *) 0x0
row = (apr_dbd_row_t *) 0x0
mirrors =
mirrors_same_prefix =
mirrors_same_as =
mirrors_same_country =
mirrors_fallback_country =
mirrors_same_region =
mirrors_elsewhere =
form_lookup = (const char *()(request_rec *, const char *)) 0xb710ad30
<__cxa_finalize@plt+416>
cmp_mirror_best =
find_best =
dbd = (ap_dbd_t *) 0x0
magnet =
found_in =
(gdb)

(gdb) l
1938 return HTTP_NOT_FOUND;
1939 }
1940 }
1941
1942
1943 statement = apr_hash_get(dbd->prepared, scfg->query_label, APR_HASH_KEY_STRING);
1944
1945 if (statement == NULL) {
1946 ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
1947 "[mod_mirrorbrain] Could not get prepared statement labelled
'%s'",

(gdb) p dbd
$1 = (ap_dbd_t *) 0x0

            ^^^^^^^^^^

Earlier in the code, a failed connection is handled like this:

 ap_dbd_t *dbd = mb_dbd_acquire_fn(r);
  if (dbd == NULL) {
      ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
              "[mod_mirrorbrain] Error acquiring database connection");
      if (apr_is_empty_array(cfg->fallbacks)) {
          setenv_give(r, "file");
          return DECLINED; /* fail gracefully */
      }
  }
  debugLog(r, cfg, "Successfully acquired database connection.");

This means that in case of fallback configuration the mirrorbrain handler continues running,
and will later try to use the database connection (*dbd), which is NULL and causes the crash
in the line

statement = apr_hash_get(dbd->prepared, scfg->query_label, APR_HASH_KEY_STRING);

This suspicion is backed up by the fact that Apache's error log shows "[mod_mirrorbrain]
Error acquiring database connection" at the same times where the crashes happened.

msg324 (view) Author: poeml Date: 2011-02-23.21:36:10

I commmitted an untested fix to svn trunk:

http://svn.mirrorbrain.org/viewvc/mirrorbrain?view=revision&revision=8238
http://svn.mirrorbrain.org/viewvc/mirrorbrain?view=revision&revision=8239

ToDo: test it; release it.

msg347 (view) Author: poeml Date: 2012-02-20.23:59:15

In 2.16.0.

History
         Date         User  Action              Args
2012-02-20 23:59:15 poeml set    status: testing -> resolved
                                   messages: + msg347
2011-02-23 21:36:18 poeml set    status: in-progress -> testing
2011-02-23 21:36:10 poeml set    messages: + msg324
2011-02-23 20:50:35 poeml create

(end of migrated issue)