-
-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ST: Reliance on EAGAIN scheduling results in significant timer error in RTC scenarios. #2194
Comments
SRS 4.0.71, when the system is under heavy load, the timer error is significant. When setting a 20ms timer, there is only a 50% chance that the trigger interval will be within 20-25ms, and in the worst case, it can be triggered at 160ms. Use srs-bench to simulate 600 streaming channels.
|
Adjust the priority of the timer coroutine and put it at the head of the running queue, which means executing the timer first. Reference: ossrs/state-threads@9fe8cfe The timer is placed at the end of the RunQ, which means executing the IO coroutine first and then the timer coroutine.
cpu=48.38%, timer=39,0,0, io=47,44,0,0,10669,109, epoll=84,3,3,25, sched=44,27,4,1,1,1,3,0,0, clock=1,13,10,4,1,1,5,1,0, co=7792,10755,84
cpu=72.62%, timer=39,0,0, io=47,44,0,0,10669,109, epoll=84,3,3,25, sched=44,27,4,1,1,1,3,0,0, clock=1,13,10,4,1,1,5,1,0, co=7792,10755,84
cpu=66.56%, timer=43,0,0, io=51,47,0,0,8505,322, epoll=296,20,20,19, sched=280,9,1,1,2,1,1,0,0, clock=1,21,9,2,1,1,3,0,0, co=7792,8469,296
cpu=61.17%, timer=43,0,0, io=51,47,0,0,8505,322, epoll=296,20,20,19, sched=280,9,1,1,2,1,1,0,0, clock=1,21,9,2,1,1,3,0,0, co=7792,8469,296
cpu=61.40%, timer=43,0,0, io=52,47,0,0,9505,250, epoll=253,18,18,23, sched=232,15,1,1,2,1,1,0,0, clock=1,21,8,3,2,1,3,1,0, co=7792,9478,253
cpu=50.31%, timer=43,0,0, io=52,47,0,0,9505,250, epoll=253,18,18,23, sched=232,15,1,1,2,1,1,0,0, clock=1,21,8,3,2,1,3,1,0, co=7792,9478,253 The timer is placed at the beginning of the RunQ, which means executing the timer coroutine first and then the IO coroutine.
cpu=47.32%, timer=36,0,0, io=42,39,0,0,11200,0, epoll=64,1,1,24, sched=25,26,6,2,1,1,2,1,0, clock=0,6,16,4,3,1,1,1,0, co=7980,11279,64
cpu=57.00%, timer=36,0,0, io=42,39,0,0,11200,0, epoll=64,1,1,24, sched=25,26,6,2,1,1,2,1,0, clock=0,6,16,4,3,1,1,1,0, co=7980,11279,64
cpu=64.40%, timer=42,0,0, io=51,47,0,0,9104,315, epoll=219,16,16,23, sched=194,20,1,1,1,1,2,1,0, clock=1,18,12,3,1,0,2,1,0, co=7980,9152,219
cpu=66.20%, timer=42,0,0, io=51,47,0,0,9104,315, epoll=219,16,16,23, sched=194,20,1,1,1,1,2,1,0, clock=1,18,12,3,1,0,2,1,0, co=7980,9152,219
cpu=62.95%, timer=41,0,0, io=51,46,0,0,10022,257, epoll=256,16,16,19, sched=231,20,1,1,1,1,2,0,0, clock=1,16,10,5,2,1,2,0,0, co=7980,9944,256
cpu=46.91%, timer=41,0,0, io=51,46,0,0,10022,257, epoll=256,16,16,19, sched=231,20,1,1,1,1,2,0,0, clock=1,16,10,5,2,1,2,0,0, co=7980,9944,256 The above data is for a small number of timers, one stream push, and 4000 clients playing. From the results, there is no difference because whether the timer is executed first or executed later, there is essentially no difference. This is because within one scheduling cycle, all coroutines are placed in the RunQ and executed together. Therefore, the timer has an equal chance and time for execution. However, executing the timer first allows us to have the opportunity to let other coroutines voluntarily yield their time slices. This allows the timer to have more opportunities for execution, enabling it to be executed multiple times within one scheduling cycle and reducing the error of the timer.
|
One important factor contributing to timer error is the busy file descriptor, for example:
We previously modified ST and moved the timer to the front, so the sequence below is as follows:
If we improve it, in the IO coroutine, if we proactively interrupt the time slice to check the timer, and execute the timer if it expires, then the sequence will be like this:
Improved strategy:
Here are the test results. Similar to above, there are 4000 RTMP players. It can be seen that the timers are almost always within 25ms, with about 50 timer events per second.
Comparing with the data before improvement, there were only 41 timer events per second (normally should be 50). Before the improvement, only the Timer was placed at the head of the RunQ.
cpu=47.32%, timer=36,0,0, io=42,39,0,0,11200,0, epoll=64,1,1,24, sched=25,26,6,2,1,1,2,1,0, clock=0,6,16,4,3,1,1,1,0, co=7980,11279,64
cpu=57.00%, timer=36,0,0, io=42,39,0,0,11200,0, epoll=64,1,1,24, sched=25,26,6,2,1,1,2,1,0, clock=0,6,16,4,3,1,1,1,0, co=7980,11279,64
cpu=64.40%, timer=42,0,0, io=51,47,0,0,9104,315, epoll=219,16,16,23, sched=194,20,1,1,1,1,2,1,0, clock=1,18,12,3,1,0,2,1,0, co=7980,9152,219
cpu=66.20%, timer=42,0,0, io=51,47,0,0,9104,315, epoll=219,16,16,23, sched=194,20,1,1,1,1,2,1,0, clock=1,18,12,3,1,0,2,1,0, co=7980,9152,219
cpu=62.95%, timer=41,0,0, io=51,46,0,0,10022,257, epoll=256,16,16,19, sched=231,20,1,1,1,1,2,0,0, clock=1,16,10,5,2,1,2,0,0, co=7980,9944,256
cpu=46.91%, timer=41,0,0, io=51,46,0,0,10022,257, epoll=256,16,16,19, sched=231,20,1,1,1,1,2,0,0, clock=1,16,10,5,2,1,2,0,0, co=7980,9944,256
|
The following is the error situation of the 20ms timer in the system when there are 3000 RTMP live streams. Before optimization, there was a significant error, with only 46 timer events per second, and only 35 timers within 25ms.
After optimization, there are 50 timer events per second, and almost all of them are within 25ms.
In order to improve the performance of the timer, high-latency coroutines actively yield. For example, if there are 10,000 iovecs to be sent in 10 batches, each batch will yield after sending 1,000 iovecs. This way, during the execution of each coroutine, the timer will be checked, ensuring that the timer does not starve. Since yield is used, it is an efficient way to relinquish the time slice, allowing the business layer to determine which coroutines have high IO latency.
|
Under RTC streaming, there is a slight difference. Instead of 50 timer events per second, there are more due to the use of multiple timers in the current RTC setup. However, it will be optimized to use only one timer in the future. So, before optimization, we can observe the distribution of timers. Out of 28 events, only 7 occur within 25ms:
After optimization, there are 50 timer events per second, and almost all of them occur within 25ms:
|
When playing RTC, yielding is mainly required when sending UDP packets. Before optimization, there were only 37 timer events per second, and only 25 of them were within 25ms.
After optimization, there are approximately 50 timer events per second, with most of them occurring within 25ms.
|
Currently, most of the timers have been optimized, but there are still a few remaining issues that need to be resolved.
|
Reference SRS High Precision, Low Error Timer: Discussing Timer Issues in High Concurrency Servers
|
ST now only switches to other coroutines when the io function encounters EAGAIN, which is undoubtedly the most efficient approach. However, when there are a large number of packages to be processed, it can lead to significant timer errors.
In the ST code, the number of scheduler switches and the time difference when entering the timer are counted. For example, during busy system times:
cpu=93.39%
- This is the CPU usage rate, which is quite high.epoll=606
- This indicates an average of 606 calls to epoll_wait per second. Note that this is an average and the distribution may not be evenly balanced.udp=60995,606
- This means that on average, recvfrom is called 60,000 times per second, with 606 of those times resulting in EAGAIN, indicating a switch to other coroutines by the scheduler.sched=536,43,4,21,1,1
- This represents the scheduler's errors (maximum possible error for timers). Out of 536 occurrences, most are within 1ms (nanosecond level), 43 are within 10ms, 4 are within 40ms, 21 are exactly 40ms, 1 is within 80ms, and 1 is within 160ms.clock=1,1,1,1,2,10,9,1
- This indicates the errors in the system's 20ms precision timer. Out of the occurrences, 1 is within 15ms, 1 is within 20ms, 1 is within 25ms, 1 is within 30ms, 2 are within 35ms, 10 are within 40ms, 9 are within 80ms, and 1 is within 160ms.Please make sure to maintain the markdown structure.
If we have a
20ms
timer enabled in an RTC (Real-Time Clock) scenario, we need to ensure that the distribution ofclock
is within the range of 15 to 25ms. This is necessary to guarantee the accuracy of this timer.Please make sure to maintain the markdown structure.
If we have multiple timers at the 20ms level enabled, there will be cumulative errors (switching to the execution of different timers will affect the next timer).
Please make sure to maintain the markdown structure.
In a live streaming scenario with a 5-second delay, it is not important, but in an RTC scenario where timers require an error within 20ms, it is a critical issue.
Please make sure to maintain the markdown structure.
Approach to the Solution
Please make sure to maintain the markdown structure.
Currently, I am considering the following approach to the solution. If anyone has a better plan, please leave a comment in the comment section.
Make sure to maintain the markdown structure.
TRANS_BY_GPT3
The text was updated successfully, but these errors were encountered: