EM Eats Up CPU to 100% #428

Closed
ben-han-cn opened this Issue Apr 2, 2013 · 2 comments

Comments

Projects
None yet
3 participants

When using thin as the web server, we found that if we change the system time to earlier time,
typically one or two hours ago, EM will eat up the cpu cycle.

We do some profile, and find that function run_machine costs a lot of time. So we dig into
the event machine code. Finally, we found the following piece of code in the c++ extension:

void EventMachine_t::_DispatchHeartbeats()
{
    while (true) {
        multimap<uint64_t,EventableDescriptor*>::iterator i = Heartbeats.begin();
        if (i == Heartbeats.end())
            break;
        if (i->first > MyCurrentLoopTime)
            break;

        EventableDescriptor *ed = i->second;
        ed->Heartbeat();
        QueueHeartbeat(ed);
    }    
}

MyCurrentLoopTime is the time before we change the system time, but the next heart beat is calculated based on
real time which is the earlier time, so the while loop will run for a quit long time and consume all the cpu resource.

Contributor

tmm1 commented Apr 2, 2013

diff --git a/ext/em.cpp b/ext/em.cpp
index 670da31..425a8ba 100644
--- a/ext/em.cpp
+++ b/ext/em.cpp
@@ -382,15 +382,27 @@ EventMachine_t::_DispatchHeartbeats

 void EventMachine_t::_DispatchHeartbeats()
 {
+   // Store the first processed heartbeat descriptor and bail out if
+   // we see it again. This fixes an infinite loop in case the system time
+   // is changed out from underneath MyCurrentLoopTime.
+   const EventableDescriptor *head = NULL;
+
    while (true) {
        multimap<uint64_t,EventableDescriptor*>::iterator i = Heartbeats.begin();
        if (i == Heartbeats.end())
            break;
        if (i->first > MyCurrentLoopTime)
            break;
+
        EventableDescriptor *ed = i->second;
+       if (ed == head)
+           break;
+
        ed->Heartbeat();
        QueueHeartbeat(ed);
+
+       if (head == NULL)
+           head = ed;
    }
 }

Contributor

garbagecat commented Apr 2, 2013

Wow, interesting bug.

On Apr 2, 2013, at 1:48 AM, Benjamin Han wrote:

When using thin as the web server, we found that if we change the system time to earlier time,
typically one or two hours ago, EM will eat up the cpu cycle.

We do some profile, and find that function run_machine costs a lot of time. So we dig into
the event machine code. Finally, we found the following piece of code in the c++ extension:

void EventMachine_t::_DispatchHeartbeats()
{
while (true) {
multimap::iterator i = Heartbeats.begin();
if (i == Heartbeats.end())
break;
if (i->first > MyCurrentLoopTime)
break;

EventableDescriptor *ed = i->second;
ed->Heartbeat();
QueueHeartbeat(ed);

}
}

MyCurrentLoopTime is the time before we change the system time, but the next heart beat is calculated based on
real time which is the earlier time, so the while loop will run for a quit long time and consume all the cpu resource.


Reply to this email directly or view it on GitHub.

tmm1 closed this in 1427a2c Apr 2, 2013

@ibc ibc added a commit to ibc/EventMachine-LE that referenced this issue Apr 2, 2013

@ibc ibc Fix bug when the system time is changed to earlier time (bug #428 in …
…EM).
a5319ff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment