Skip to content

Commit

Permalink
fix race crash for accepting new connections
Browse files Browse the repository at this point in the history
Inspired by a patch by Kazuki Ohta.

Summary from a mail sent to the list by Shigeki:
[Example Scenario]
 1. throw alot clients (well over connection limit) to connect to
memcached.
 2. memcached's file descriptors reaches maximum setting
 3. main thread calls accept_new_conns(false) to stop polling sfd
 4. main thread's event_base_loop stops accepting incoming request
 5. main thread stops to acceess main_base at this point
 6. a client disconnects
 7. worker thread calls accept_new_conns(true) to start polling sfd
 8. accept_new_conns uses mutex to protect main_base's race condition
 9. worker thread starts loop with listen_conn
10. worker thread calls update_event with first conn
11. after first update_event(), main thread start polling sfd and
    starts to access main_base <- PROBLEM
12. Worker thread continues to call update_event() with second conn

At this point, worker thread and main thread both acccess and modify
main_base.
---

The original patch coupled polling with the once-per-second clock timer. My
patch creates a 10ms poller which kicks off after the listener is disabled.
Switching for a conditional would be too much rewiring for 1.4, as 1.6 solves
this in a better way.
  • Loading branch information
dormando committed Jul 12, 2011
1 parent 0d45e55 commit ac93972
Showing 1 changed file with 26 additions and 1 deletion.
27 changes: 26 additions & 1 deletion memcached.c
Expand Up @@ -120,6 +120,27 @@ enum transmit_result {

static enum transmit_result transmit(conn *c);

/* This reduces the latency without adding lots of extra wiring to be able to
* notify the listener thread of when to listen again.
* Also, the clock timer could be broken out into its own thread and we
* can block the listener via a condition.
*/
static volatile bool allow_new_conns = true;
static struct event maxconnsevent;
static void maxconns_handler(const int fd, const short which, void *arg) {
struct timeval t = {.tv_sec = 0, .tv_usec = 10000};

if (allow_new_conns == false) {
/* reschedule in 10ms if we need to keep polling */
evtimer_set(&maxconnsevent, maxconns_handler, 0);
event_base_set(main_base, &maxconnsevent);
evtimer_add(&maxconnsevent, &t);
} else {
evtimer_del(&maxconnsevent);
accept_new_conns(true);
}
}

#define REALTIME_MAXDELTA 60*60*24*30

/*
Expand Down Expand Up @@ -509,7 +530,9 @@ static void conn_close(conn *c) {

MEMCACHED_CONN_RELEASE(c->sfd);
close(c->sfd);
accept_new_conns(true);
pthread_mutex_lock(&conn_lock);
allow_new_conns = true;
pthread_mutex_unlock(&conn_lock);
conn_cleanup(c);

/* if the connection has big buffers, just free it */
Expand Down Expand Up @@ -3363,6 +3386,8 @@ void do_accept_new_conns(const bool do_accept) {
stats.accepting_conns = false;
stats.listen_disabled_num++;
STATS_UNLOCK();
allow_new_conns = false;
maxconns_handler(0, 0, 0);
}
}

Expand Down

0 comments on commit ac93972

Please sign in to comment.