Meaning of CPU time in ProcessCounters

[amanjeetsingh150]

Hey,

I am trying to understand why the CPU time is inferred as the sum of user time and system time through getrusage here in ProcessCounters.cpp .

CPU time of process to me is the time when the process is executing instructions on CPU and is not idle. Ideally, the code should not include the time when the CPU is idle due to waiting on I/O or due to lock.

Resources: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html

I ran an experiment reading large CSV files which is an IO-bound task (CMIIW) and this number (rusageStats.ru_utime + rusageStats.ru_stime) starts incrementing quickly and according to the definition, it should not grow with quickly large numbers as the CPU will be idle for this process.

Can you suggest some experiments which I can run on my emulators or device to validate that this time will not increment if I keep my CPU idle or stalled due to some external resource?

[amanjeetsingh150]

Hey, any insights on above? @ricardorey10 @BurntBrunch

[BurntBrunch]

There are a couple of different ideas that are worth disambiguating here:

• CPU utilization, i.e. what top(1) reports. Brendan is right to call this non-idle time as it (and its cousin load average from uptime(1)) revolve around the depth of the run queue on each core and the idle task being scheduled.
• Scheduled-in time - wall clock time that a thread has spent running on a core.
• Instructions/Cycles/CPU Frequency/etc - the actual performance characteristics that we care about and don't have access to :)

What CPU time means across the kernel APIs is "scheduled-in" time.
In fact, the proc(5) man page is pretty explicit about this:

... about `/proc/[pid]/stat`: ...
(14) utime  %lu
  Amount of time that this process has been scheduled in user mode, measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).  This includes guest time, guest_time (time spent running a virtual  CPU, see below), so that applications that are not aware of the guest time field do not lose that time from their calculations.

(15) stime  %lu
  Amount of time that this process has been scheduled in kernel mode, measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).

Notice the usage of "scheduled." It specifically means that the thread was on a core, executing code. I/O would not count against this time and futex/semaphore/etc-using locks would not either (both deschedule the thread while the I/O is completing or the lock is locked). Spin locks would of course count here.

You can see this in the getrusage code too:

1. getrusage reads the cpu time from the task group object - https://github.com/torvalds/linux/blob/v4.10/kernel/sys.c#L1580-L1595
2. If you follow the rabbit hole from there, you'll see that it's mostly about dealing with what the scheduler has precomputed and any vtime that has elapsed since then. https://github.com/torvalds/linux/blob/v4.10/kernel/sched/cputime.c#L854 and onwards.

Now, the question is, "how is scheduled-in time useful?"

It turns out that people can reason pretty well about this time. It corresponds to their code, it matches their intuition of "CPU time" well enough to optimize hot paths, we can collect stack trace samples running on this same clock, and overall is a reasonable approximation of what the device was actually running. It is an approximation - we can't account for changing core frequencies, task migration cache busting, or just normal memory stalls - but a useful one.

It's also important to note that you often need both wall-time and cpu-time measurements in the same trace. We make heavy use of instrumented code sections (Trace.beginSection etc) and even capture some I/O counters.

Can you suggest some experiments which I can run on my emulators or device to validate that this time will not increment if I keep my CPU idle or stalled due to some external resource?

Sure. Create a standalone binary that opens a socket and waits for a connection. Notice that CPU time (as reported by /proc/<pid>/stat) doesn't change while waiting.

What you might be seeing in your experiment is that sampling the counters uses CPU time. Unfortunately, we don't have access to APIs that notify us when the values change, so we have to poll periodically. However, even in this case, what you should see is that the thread-specific CPU time counters only move up on the profilo threads and not the thread where your experiment is being conducted.

Hope this helps!

[amanjeetsingh150]

Thank you so much @BurntBrunch this gives good insights. Closing this now.