New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crash when database connection fails in presence of fallback #84

Closed
poeml opened this Issue Jun 5, 2015 · 0 comments

Comments

Projects
None yet
1 participant
@poeml
Owner

poeml commented Jun 5, 2015

                                                                                            [          ]

Issue migrated (2015-06-05) from old issue tracker http://mirrorbrain.org/issues/issue84

Title    crash when database connection fails in presence of fallback
              mirrors
 Priority   bug                       Status            resolved
Superseder                           Nosy List          poeml
Assigned To poeml                    Keywords

msg323 (view) Author: poeml Date: 2011-02-23.20:50:32

I saw a crash that seems to occur when the database connection fails (for any reason) and
fallback mirrors are configured:

(gdb) thread apply all bt
....

(gdb) thread 24
[Switching to thread 24 (Thread 29704)]#0 0xffffe422 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe422 in __kernel_vsyscall ()
#1 0xb738ccb6 in kill () from /lib/libc.so.6
#2 0xb7795e06 in sig_coredump (sig=11) at mpm_common.c:1231
#3
#4 0xb70f594c in mb_handler (r=0xb798f540) at mod_mirrorbrain.c:1943
#5 0xb778c03d in ap_run_handler (r=0xb798f540) at config.c:158
#6 0xb778fbb4 in ap_invoke_handler (r=0xb798f540) at config.c:376
#7 0xb779c641 in ap_process_request (r=0xb798f540) at http_request.c:282
#8 0xb77990d8 in ap_process_http_connection (c=0xb798b6e0) at http_core.c:190
#9 0xb779449d in ap_run_process_connection (c=0xb798b6e0) at connection.c:43
#10 0xb77a2c2c in worker_thread (thd=0xb77fd538, dummy=0xb78235d0) at worker.c:544
#11 0xb7542b56 in ?? () from /usr/lib/libapr-1.so.0
#12 0xb74ca1b5 in start_thread () from /lib/libpthread.so.0
#13 0xb743362e in clone () from /lib/libc.so.6

(gdb) up
...

(gdb) l
1938 return HTTP_NOT_FOUND;
1939 }
1940 }
1941
1942
1943 statement = apr_hash_get(dbd->prepared, scfg->query_label, APR_HASH_KEY_STRING);
1944
1945 if (statement == NULL) {
1946 ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
1947 "[mod_mirrorbrain] Could not get prepared statement labelled
'%s'",

(gdb) i lo
cfg = (mb_dir_conf ) 0xb7995950
ptr =
uri =
filename = 0xb79964d9 "stable/3.3.0/OOo_3.3.0_MacOS_x86_install_en-US.dmg"
basename = 0xb79964e6 "OOo_3.3.0_MacOS_x86_install_en-US.dmg"
mirror_base = 0xb7996480 "/srv/ooo"
realfile =
yum = (yumdir_t *) 0x0
clientip = 0xb798bb60 "128.232.241.11"
clientaddr =
query_country = 0x0
query_asn = 0x0
fakefile = 0 '\0'
only_hash = 0 '\0'
rep = 0
rep_ext = 0x0
meta_negotiated = 0 '\0'
continent_code = 0xb70b92c7 "EU"
country_code = 0xb70b8bc7 "GB"
country_name = 0xb70b8078 "United Kingdom"
slat =
slng =
lat = 30.4797993
lng = -87.865799
state_id =
state_name =
as = 0xb71003dd "--"
prefix = 0xb71003dd "--"
i = -1215364960
mirror_cnt =
len = 20
nr = 3056701544
new =
mirror =
mirrorp =
chosen =
hashbag = (hashbag_t *) 0x0
rv =
res = (apr_dbd_results_t *) 0x0
row = (apr_dbd_row_t *) 0x0
mirrors =
mirrors_same_prefix =
mirrors_same_as =
mirrors_same_country =
mirrors_fallback_country =
mirrors_same_region =
mirrors_elsewhere =
form_lookup = (const char *(
)(request_rec *, const char *)) 0xb710ad30
<__cxa_finalize@plt+416>
cmp_mirror_best =
find_best =
dbd = (ap_dbd_t *) 0x0
magnet =
found_in =
(gdb)

(gdb) l
1938 return HTTP_NOT_FOUND;
1939 }
1940 }
1941
1942
1943 statement = apr_hash_get(dbd->prepared, scfg->query_label, APR_HASH_KEY_STRING);
1944
1945 if (statement == NULL) {
1946 ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
1947 "[mod_mirrorbrain] Could not get prepared statement labelled
'%s'",

(gdb) p dbd
$1 = (ap_dbd_t *) 0x0

            ^^^^^^^^^^

Earlier in the code, a failed connection is handled like this:

 ap_dbd_t *dbd = mb_dbd_acquire_fn(r);
  if (dbd == NULL) {
      ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
              "[mod_mirrorbrain] Error acquiring database connection");
      if (apr_is_empty_array(cfg->fallbacks)) {
          setenv_give(r, "file");
          return DECLINED; /* fail gracefully */
      }
  }
  debugLog(r, cfg, "Successfully acquired database connection.");

This means that in case of fallback configuration the mirrorbrain handler continues running,
and will later try to use the database connection (*dbd), which is NULL and causes the crash
in the line

statement = apr_hash_get(dbd->prepared, scfg->query_label, APR_HASH_KEY_STRING);

This suspicion is backed up by the fact that Apache's error log shows "[mod_mirrorbrain]
Error acquiring database connection" at the same times where the crashes happened.

msg324 (view) Author: poeml Date: 2011-02-23.21:36:10

I commmitted an untested fix to svn trunk:

http://svn.mirrorbrain.org/viewvc/mirrorbrain?view=revision&revision=8238
http://svn.mirrorbrain.org/viewvc/mirrorbrain?view=revision&revision=8239

ToDo: test it; release it.

msg347 (view) Author: poeml Date: 2012-02-20.23:59:15

In 2.16.0.

History
         Date         User  Action              Args
2012-02-20 23:59:15 poeml set    status: testing -> resolved
                                   messages: + msg347
2011-02-23 21:36:18 poeml set    status: in-progress -> testing
2011-02-23 21:36:10 poeml set    messages: + msg324
2011-02-23 20:50:35 poeml create

(end of migrated issue)

@poeml poeml added bug resolved labels Jun 5, 2015

@poeml poeml closed this Jun 5, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment