Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Clone this wiki locally
Sources of CPU interference in core Linux code
Linux version: 3.3
CPU interference: For the the purpose of this document, CPU interference is defined as any case where kernel management functions will take CPU time from a pure CPU bound user task making no system call and running pinned to an isolated CPU with no other contending user tasks. Specifically, any kernel activity that occurred as a result of the the specific task activity is not considered interference. However, kernel activity on the specific CPU happening as a result of work of other tasks or the general kernel infrastructure not related a specific task are considered interference.
- fs/buffer.c - Global IPI for each LRU drain 
- kernel/hrtimer - Global IPI for setting high res-timer when system clock is changed 
- kernel/profile.c - double flip buffer management, used during profiling only 
- kernel/rcutree.c - rcu_barrier() implementation uses a global IPI to queue an RCU callback on every CPU
- mm/page_alloc.c - global IPI for draining all per-cpu pages  
- mm/slab.c - global IPI during per cpu cache drain and tuning 
- mm/slub.c - global IPI during per cpu caches drain, called when destroying a cache (also does an rcu_barrier) 
- net/core/dev.c - global IPI to flush backlog during net device unregister (also does an rcu_barrier) 
Global work queue scheduling
- mm/swap.c - global work scheduled on each cpu for draining pagevecs to LRU lists 
- mm/slab.c - delayed work scheduled one per second to reap per cpu slab cache
- mm/vmstat.c - delayed work scheduled one per second for pcp cache draining and VM statistics
Global kthread scheduling
- kernel/stop_machine.c - anything using stop_machine mechanism, such as
- synchronize_..._expedited 
- module unload 
- cpu hot-unplug 
- text_poke (used for kprobes) 
- kernel/time/tick-sched.c - the scheduler tick runs the scheduler, run queue time keeping, timer management, RCU 
- kernel/time/clocksource.c - the clocksource_watchdog registers a timer on each CPU in a cyclic manner 
- Dealt with in Reduce cross CPU IPI interference patch set (https://lkml.org/lkml/2012/1/8/109). This patch set is now in Linus 3.4-rc0 tree.
- Controlled outside events, most of which not normally done during normal production run time (maybe with the exception of module load/unload).
- Since slab and slub are mutually exclusive it is enough to deal with one of them.
- Made a very rare event by the Mel Gorman's patch to not do a global drain for direct reclaim path of memory allocator (https://lkml.org/lkml/2012/1/11/93).
- Might be dealt with by the effort to get rid of pagevecs (see: https://lkml.org/lkml/2012/1/4/376) or we can do schedule_on_each_cpu_mask(...).
- Dealt with by Frederic Weisbecker NOHZ cpuset patchset (see: http://lwn.net/Articles/455044/)
- Paul McKenny now has "Make synchronize_sched_expedited() avoid IPIing idle CPUs" on his to do list: http://kernel.org/pub/linux/kernel/people/paulmck/rcutodo.html.
- Only relevant if your clock source is not stable. Can be turned off by a CONFIG option, which is currently hard coded to true for x86. This patch allows is to be turned off: https://lkml.org/lkml/2012/3/27/193.
Thank you Frederic Weisbecker, Peter Zijlstra, Paul E. McKenney, Christoph Lameter for contrinuting to this list!