Memory leak in Deno (on Deno versions >= 1.31.0)

[quentinadam]

The below very simple program will ostensibly (at a rate of ~1MB/second) leak memory in Deno (only on Deno versions >= 1.31.0). To reproduce, please add a hello.txt file with a random content such as Hello! at the root and run the below program:

const file = new URL('./hello.txt', import.meta.url).pathname;
while (true) {
  Deno.readFileSync(file);
}

The memory leak only happens on Deno versions >= 1.31.0. It does not happen on Deno versions <= 1.30.3.
The memory leak is more significant with shorter files.

Here is a slightly longer version of the program that will output the memory usage every second :

console.log('Deno version', Deno.version.deno);

const file = new URL('./hello.txt', import.meta.url).pathname;

let timestamp = new Date();

while (true) {
  if (Date.now() >= timestamp.valueOf()) {
    const bytes = Deno.memoryUsage().rss;
    console.log(timestamp.toISOString(), Math.floor(bytes / (1024 * 1024) * 10) / 10);
    timestamp = new Date(timestamp.valueOf() + 1000);
  }
  Deno.readFileSync(file);
}

I have also created a repository that showcases the issue: https://github.com/quentinadam/deno-memory-leak.

[dsherret]

I wasn't initially able to reproduce this, but then I used the file you had with just "Hello!" in it and I was able to. This is not reproducible on Windows it seems.

[quentinadam]

Yes, the memory leak is more obvious with small files (I should have mentioned it). Although, I think the memory leak is also present with larger files too. I have the impression the memory leak is due to the loop, hence why the memory leak is more significant when the loop is faster (when reading smaller files).

[quentinadam]

The memory leak also happens with fetch. To reproduce, please first run a simple HTTP server (like the one provided on https://deno.land/manual@v1.31.1/examples/http_server) and run the following code:

console.log('Deno version', Deno.version.deno);

let timestamp = new Date();

while (true) {
  if (Date.now() >= timestamp.valueOf()) {
    const bytes = Deno.memoryUsage().rss;
    console.log(timestamp.toISOString(), Math.floor(bytes / (1024 * 1024) * 10) / 10);
    timestamp = new Date(timestamp.valueOf() + 1000);
  }
  const response = await fetch('http://localhost:8080');
  await response.text();
}

Here is a chart of the memory usage over the first 10 minutes when I run the program on MacOS ARM64 (I experience similar results on Linux x86-64):

[ry]

V8 is quite lazy when it comes to GC. Try --v8-flags=--expose-gc and call gc() in the loop.

There could very well be a memory leak, but this plot doesn't say much.

[dsherret]

@ry it still runs out of memory, but at a slower pace. I didn't test fetch though. For the fetch test, it should be run until it runs out of memory because something might be caching or utilizing system memory.

[quentinadam]

@dsherret, when I run the program (the one reading the file) with --v8-flags=--expose-gc and by calling gc()in the loop, it does seem to somewhat solve the memory leak issue. Here is a chart of the memory usage over time:

However, having to resort to calling gc() does not feel right and it is still odd that:

• None of this behaviour happens before v1.31.0
• Without explicitly calling gc() explicitly, the memory usage of the program will grow continuously until memory exhaustion, at which point the program will be killed.

[naegelin]

I can confirm we are seeing this also on long running Deno processes in Deno v1.32.1 .

After some time Deno exits with Cannot grow ExternalPointerTable The problem does not occur prior to 1.31.0 . We are using both fetch and readFileSync.

[aapoalas]

Do you get any sort of backtrace from the crash?

[lucacasonato]

Bisected, and unfortunately it's this:

65500f36e870b4ada3996b06aa287e30177d21a3 is the first bad commit
commit 65500f36e870b4ada3996b06aa287e30177d21a3
Author: Bartek Iwańczuk <biwanczuk@gmail.com>
Date:   Tue Feb 7 13:36:41 2023 +0100

    chore: upgrade rusty_v8 to 0.62.2 (#17604)
    
    Co-authored-by: Bert Belder <bertbelder@gmail.com>

 Cargo.lock                                         |   4 ++--
 Cargo.toml                                         |   2 +-
 .../run/worker_close_in_wasm_reactions.js.out      |   1 +
 core/icudtl.dat                                    | Bin 10454784 -> 10541264 bytes
 core/inspector.rs                                  |  18 ++++++++++++++++++
 core/ops_builtin_v8.rs                             |   8 ++++----
 core/runtime.rs                                    |   4 ++--
 7 files changed, 28 insertions(+), 9 deletions(-)

[bartlomieju]

We'll look into the V8 upgrade problem this week.

[levex]

https://bugs.chromium.org/p/v8/issues/detail?id=13640&q=garbage%20collection&can=2

and

https://chromium.googlesource.com/v8/v8/+/46e5212fdc8193cb410fde6a34fd3377c8b99c5e

From V8

[bartlomieju]

We're floating a V8 patch to verify if this is a V8 regression.

[lucacasonato]

@naegelin Do you have a backtrace / crashdump for the error you are seeing? Is the code that is causing Deno to crash public? If not, are you able to share it with us privately (if so, you can email me at luca@deno.com).

[naegelin]

Hey @lucacasonato unfortunately I cannot share the code. I have restarted a process with 1.32.1 and will try to capture the crashdump. It can take several hours so I'll report back when I have something.

[jmordica]

👀

[sergeysolovev]

@bartlomieju

Thank you for the fix! This is not yet included in v1.32.5, correct?

[bartlomieju]

No, it's not. I will be released later this week 👍