with node16 I see performance of async/await is worse than node 12 #38339
Labels
performance
Issues and PRs related to the performance of Node.js.
Comments
|
The test code (from https://fibjs.org/en/docs/guide/about.md.html [Embrace high energy] section) var count = 1000;
async function test_async(n) {
if (n == count)
return;
await test_async(n + 1);
}
function test_callback(n, cb) {
if (n == count)
return cb();
test_callback(n + 1, cb);
}
function test_sync(n) {
if (n == count)
return;
test_sync(n + 1);
}
async function test() {
console.time("async");
await test_async(0);
console.timeEnd("async");
console.time("callback");
test_callback(0, () => {
console.timeEnd("callback");
});
console.time("sync");
test_sync(0);
console.timeEnd("sync");
}
test(); //actually I also add a python timeit like code to repeat the test() |
Ayase-252
added
the
performance
|
What are the results you are seeing? Here are the results on my machine (macOS 11.2, x86_64): I assume "less is better", right? |
|
Your result is different from mine, I run each test 10 times and several runs, following is one of my results. Node 16 is always the worst |
|
My numbers also show improvement (macOS 10.15.7, x86_64): Node 12.22: Node 14.16: Node 16: |
|
@Linkgoron @aduh95 I found you guys just run test once, try to run them several times, as simple as |
|
In my machine, it varies greatly in different runs. Could you provide some statistics like average and variance of run time in your tests? It would help us to identify whether there is performance regression. |
|
@Ayase-252 will do. But let me find more machines to repeat my test first. |
|
@Linkgoron @Ayase-252 @aduh95 if you just run my test once, node 16 indeed performs best. I have the similar result like yours. But if you run test more than once, say 10 or 15 time, node 16 becomes worst. I initially use some home-made |
|
I've made a benchmark script that uses Node.js benchmarking tool: 'use strict';
const common = require('../common.js');
const bench = common.createBenchmark(main, {
count: [1, 10, 100, 1_000, 5_000],
test: ['async', 'callback', 'sync'],
});
async function test_async(n) {
if (n === 0) return;
await test_async(n - 1);
}
function test_callback(n, cb) {
if (n === 0) return cb();
test_callback(n - 1, cb);
}
function test_sync(n) {
if (n === 0) return;
test_sync(n - 1);
}
function main({ count, test }) {
bench.start();
switch (test) {
case 'async':
test_async(count).then(() => bench.end(count));
break;
case 'callback':
test_callback(count, () => bench.end(count));
break;
case 'sync':
test_sync(count);
bench.end(count);
break;
}
}I put it in $ /usr/local/bin/node --version
v16.0.0
$ /usr/local/Cellar/node@14/14.16.0_1/bin/node --version
v14.16.0
$ node benchmark/compare.js --new /usr/local/bin/node --old /usr/local/Cellar/node@14/14.16.0_1/bin/node -- async | Rscript benchmark/compare.R
[00:00:56|% 100| 1/1 files | 60/60 runs | 15/15 configs]: Done
confidence improvement accuracy (*) (**) (***)
async/async.js test='async' count=1 *** 9.85 % ±5.55% ±7.38% ±9.60%
async/async.js test='async' count=10 * 6.40 % ±5.22% ±6.96% ±9.06%
async/async.js test='async' count=100 6.20 % ±6.47% ±8.61% ±11.22%
async/async.js test='async' count=1000 *** -10.17 % ±5.22% ±6.95% ±9.05%
async/async.js test='async' count=5000 ** 6.30 % ±4.56% ±6.07% ±7.89%
async/async.js test='callback' count=1 *** 15.90 % ±4.87% ±6.49% ±8.45%
async/async.js test='callback' count=10 *** 17.07 % ±4.87% ±6.48% ±8.43%
async/async.js test='callback' count=100 *** 13.82 % ±5.64% ±7.52% ±9.82%
async/async.js test='callback' count=1000 *** 10.43 % ±5.63% ±7.49% ±9.76%
async/async.js test='callback' count=5000 -4.47 % ±5.50% ±7.32% ±9.53%
async/async.js test='sync' count=1 *** 18.13 % ±5.25% ±6.99% ±9.11%
async/async.js test='sync' count=10 *** 16.84 % ±5.43% ±7.23% ±9.43%
async/async.js test='sync' count=100 *** 10.78 % ±4.64% ±6.18% ±8.04%
async/async.js test='sync' count=1000 0.70 % ±5.46% ±7.28% ±9.50%
async/async.js test='sync' count=5000 ** -8.14 % ±5.69% ±7.57% ±9.86%
Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 15 comparisons, you can thus
expect the following amount of false-positive results:
0.75 false positives, when considering a 5% risk acceptance (*, **, ***),
0.15 false positives, when considering a 1% risk acceptance (**, ***),
0.01 false positives, when considering a 0.1% risk acceptance (***)As we can see, Node.js v16.0.0 has worse performance on async calls when count is |
|
@aduh95 I tried to repeat your test with count = 500 and I found the node 16 was not much different from node 14 or 12. I start to wonder is my test valid or meaningful at all ? Back to the time when I upgraded from node 10 to node 12 I saw obvious improvement under any test. So during that time I had thought my test was valid. |
|
So I've ran more benchmarks to understand what's the trend, here are the results for the async version only:
The extreme value is -73.53 % for the value 1730 on my machine. If we graph the performance for each
@nodejs/v8 The performance decrease seems significant, it's probably worth taking a look. Complete data |
|
I notice your test is using functions from |
|
@mattwelke Node.js benchmark suite is using |
|
Ah sorry I was on mobile and couldn't see all of that comment. Someone linked me here and asked me to take a look. I see there's actually a performance regression here. |
|
Has there been any further discussion on this internally? I saw similar performance issues after deploying some production APIs on node 16 (previously node 14). Our APIs make heavy use of async/await, and immediately after the deployment was completed I saw an uptick in CPU and memory usage, along with increased response times - we ended up rolling back to 14. These APIs were running on AWS ECS (Fargate). We use the Unfortunately, I no longer have stats from that real-world scenario, but following suit with some of the benchmarks above i wrote a super basic test that just iterated through a loop and const fn = () => Promise.resolve();
const n = // 100000, 500000, 1000000, 5000000, etc...
for (let i = 0; i < n; i++) await fn();Running the test on Node 16 was consistently slower than Node 14 after n reached about ~1M (also slower than Node 17).
I know that's a silly test to run, but it does show that something is going on. How it plays out in a real-world scenario likely depends on the workload, but like I said we heavily use async/await in our applications and definitely can't upgrade to node16+ in its current state :( Btw, im on a 2018 MPB, core i9, 32gb memory, using nvm for node version switching. |
|
@rileyweber Mind sharing a repo for the benchmark you ran? I've got a different setup and I can try to reproduce it so that they have more reproductions posted here. |
|
@mattwelke I think you can just use https://fibjs.org/en/docs/guide/about.md.html to reproduce it (as when I first opened this issue) |
|
I just noticed your benchmark was only the code you posted here. So I ran it with n = 5000000. Results: node -v && time node index.js
v10.24.1
real 0m0.418s
user 0m0.411s
sys 0m0.065snode -v && time node index.js
v12.22.9
real 0m0.211s
user 0m0.223s
sys 0m0.024snode -v && time node index.js
v14.18.3
real 0m0.223s
user 0m0.206s
sys 0m0.052snode -v && time node index.js
v16.13.2
real 0m0.255s
user 0m0.270s
sys 0m0.000snode -v && time node index.js
v17.4.0
real 0m0.237s
user 0m0.232s
sys 0m0.021sMy system: AMD Ryzen 7 5800X 8-Core Processor lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal |
|
I'm not seeing much of a performance regression between 14 and 16 though once n gets a lot higher. In fact, eventually performance increases. Here's n = 50M: matt@DESKTOP-BVT2KNU:~/node-benchmark$ n 14
installed : v14.18.3 (with npm 6.14.15)
matt@DESKTOP-BVT2KNU:~/node-benchmark$ node -v && time node index.js
v14.18.3
real 0m2.038s
user 0m2.080s
sys 0m0.284s
matt@DESKTOP-BVT2KNU:~/node-benchmark$ n 16
installed : v16.13.2 (with npm 8.1.2)
matt@DESKTOP-BVT2KNU:~/node-benchmark$ node -v && time node index.js
v16.13.2
real 0m2.046s
user 0m2.063s
sys 0m0.073sAnd n = 1B: matt@DESKTOP-BVT2KNU:~/node-benchmark$ n 14
installed : v14.18.3 (with npm 6.14.15)
matt@DESKTOP-BVT2KNU:~/node-benchmark$ node -v && time node index.js
v14.18.3
real 0m37.034s
user 0m38.664s
sys 0m3.602s
matt@DESKTOP-BVT2KNU:~/node-benchmark$ n 16
installed : v16.13.2 (with npm 8.1.2)
matt@DESKTOP-BVT2KNU:~/node-benchmark$ node -v && time node index.js
v16.13.2
real 0m36.557s
user 0m36.837s
sys 0m0.344s |
|
Yea overall I agree with the sentiment that benchmarks never show the full picture - the script I wrote was really just to get an idea. I also ran the benchmarks posted higher up in the thread and saw similar results where 16 consistently performed worse with async/await than 14 (and 12). Mainly I don't want this conversation to fizzle out - I don't see anyone else talking about it on SO, etc., and I don't see related V8 posts - but these benchmarks and production workloads that I've seen show that something seems off. |
|
Makes sense. I don't have any skin in this game because we're planning on phasing out our Node.js in production right now instead of upgrade it to 16+, so I'm out. But I do hope you get some eyes on it since you say it's affecting you. I think general consensus is that small performance regressions aren't a big deal. They happen all the time, and we generally solve performance problems by just throwing more hardware at them. But for large regressions, that's a different story. Maybe if you shared stats on how badly performance worsened for you (you mentioned seeing large CPU usage increases), it would help the maintainers see the issue here. |
|
I notice node 18 was released and V8 JavaScript engine was upgraded to 10.1 so I did the test again and found node18 has significantly improved the test result I did, almost reduce the 50% of running time of nodejs 14. Actually although in my initial test node 16.0 was slower than node 14.15, I find node 16.14.2 has performed better than node 14.19.0. So I closed this issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version: 16.0.0
Platform: Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64
MacbookPro 2015 , 2.5 GHz Quad-Core Intel Core i7, 16 GB 1600 MHz DDR3
What steps will reproduce the bug?
Before I file this issue I know benchmarks never tell the whole story. But my goal here is relatively simple, I want to see the performance improvement regarding async functions and promises.
I use n to install following node version:
I run a simple performance test code using the idea from https://fibjs.org/ and write my own
timeitto repeat the test.I actually asked the question at SO https://stackoverflow.com/questions/64886560/simple-practical-example-to-see-faster-async-functions-and-promises-from-node-10 when node 14 came out.
The reason I file the issue here is I test my code against node 16. and found it 's performance is worse than 12 & 14.
The text was updated successfully, but these errors were encountered: