question: Average CPU consumption range and threshold for Node.js apps in Production

[bidipyne]

Given that Node.js provides a single threaded, asynchronous, event driven programming model which handles the I/O and CPU bound operations efficiently, through multiplexing between them, consider this program:
Client code
Server code

In short, some clients are connecting (and reconnecting) to a server and downloading some data. From server standpoint, it is addressing the needs of many clients, triggered by events in the uv loop, in a heavily concurrent manner.

The top command shows that node is utilizing more that 99% of CPU time.

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23766 bidisha  20   0  672m  14m 9828 R 99.9  0.1   4:03.43 node

In a multi-threaded server program execution environment (such as Java application server) which works with a thread-pool to handle concurrent clients, a healthy CPU consumption is considered to be less than 20% (IMO).

In node, the increased CPU consumption is understood to be root caused by increased activity by a single thread across many transactions without blocking on I/O and consuming most or all of its CPU share. And in this case, I don't see any issue, as the increased consumption is translated to useful work done.

But then, what is optimal range of CPU consumption which is considered to be healthy? And what is the threshold point beyond which one should start suspecting its consumption and start profiling for CPU? Are there guidance values available?

Thanks in advance!

[mscdex]

Is the 'top' output you've shown the result of the server code you've linked to, or is it the result of more complex application logic and the server code you linked to is different/just an example?

Also, what node version, OS, and OS/kernel version is this with?

[bidipyne]

Yes, the top output is the result of the server code.
Node version : v7.5.0
OS version: Linux 2.6.32-131.12.1.el6.x86_64

[gireeshpunathil]

I guess the CPU consumption should be viewed as a function of the throughput. For example, define a baseline for the througput in a given test scenario alongwith the CPU consumption for the baseline, and then appy the real workload, and observe how the CPU consumption is fairing against the throughput. If the throughput:CPU ratio is steady, I guess the consumption rate is immaterial.

Even if the consumption is 100%.

If backed up by throughput improvement against a baseline, this would mean that node is able to scale upto the required concurrency for the workload, and multiplexing between requests and responses without any blocking operations in between, consuming all the alloted time slot for the process, restlessly and relentlessly.

And this is a good, and ideal thing to have, for a high performant code.

But the above question has relevance in terms of performance monitoring tools - how do a tool spot an anomaly in the CPU consumption, and what is the boarder line which separates a healthy and a CPU burning code?

[joyeecheung]

In a model similar to Node.js's I would look into what the CPU consumption consists of instead of how much consumption is. As long as the event loop is not blocked and is doing meaningful work, the process is not spending too much time on GC(< 5% in my experience), and the latency & throughput are acceptable(depending on what it does), it doesn't matter how high the CPU consumption is.

[bidipyne]

Thanks! Looks like we all are on agreement.
It is the breakup of the CPU and the portion of it, which is utilized for application which matters and decides the efficiency. This would warrant an internal profiling tool as opposed to an external tool like top.

I would look into what the CPU consumption consists of instead of how much consumption is

Agree, but when do you think it is appropriate to do this check? Throughout the life cycle of application or when the CPU shoots up beyond some threshold ?

[joyeecheung]

@bidipyne From my experience and that of other teams that I know, these indexes (responsiveness of the event loop, gc time, latency or apdex, QPS/TPS...) should be monitored in production (and set alerts for), and watched when doing load testing.

[bidipyne]

While I understand the CPU percentage in isolation does not indicate anything in node other than the degree of activities in the system, it looks a grey matter for me how to gather QPS or TPS from a server side (I thought these are client side artifacts) and co-relate them with the activities in the virtual machine and defining some profiling trigger, leading to making some inferences.

Closing this issue as I don't see any more responses. Thanks @joyeecheung and @gireeshpunathil for providing vital insights!