[BUG] Memory increases when same context is used

[Romain-P]

Context:

• Playwright Version: Latest (today is 26/044/2021)
• Operating System: Linux
• Node.js version: tested on both node.js version
• Browser: chromium

Describe the bug

I'm watching full-js apps (e.g react/angular websites). I initialize one instance, 1 browser and 1 page.
I keep the page in cache & retrieving content every 2 seconds.

After 1/2hours, the memory goes crazy. I tried to reload() the page every 30 minutes. It doesn't free the memory.
Only way to free the memory is closing the page and recreating a new one.

What could be the source of this memory leak? I suppose reload() frees the javascript-vm so it must be a leak internally to the page

[mxschmitt]

Currently the objects for e.g. request/response/route get only flushed when a new context is created. For testing you typically create a new context for each test. I folded a few issues into this one of users who also faced into that, so we can better keep track of it and might find a workaround for it in the future.

[Romain-P]

I am recreating the page every 1 hour this is the only work around I have found to keep the memory stable.
Why would recreating the context actually helps? I mean doing page.close() and the re-allocating a page using broswer.newPage() is fixing the issue

[mxschmitt]

browser.newPage does internally also create a new context for you and close it once you close the page, so its basically a helper wrapper to simplify its usage.

[VikramTiwari]

Hey @mxschmitt Are there some other similar quirks that can be used to lower memory/CPU usage? I am starting to look into performance and parallelization of our tests and things like this would be helpful. Thanks!

[mxschmitt]

The Playwright test runner gets released soon, stay tuned! It handels all that for you.
Headless requires mostly less cpu/memory than headed.

See here: https://playwright.dev/docs/intro

[LDubya]

Currently the objects for e.g. request/response/route get only flushed when a new context is created

I think being able to have the page flush responses when new ones are received would be a useful feature, and along the lines of the way a typical browser handles responses. Maybe have it as a flag we can set?

While headless browsers have their origins in browser testing, and browser testing continues to be a major use case, data science is a rapidly growing field, and scraping is a major selling point for using headless browsers.

When scraping, you may not want to create an entirely new context with each data retrieval, as the cookies may be important for storing complex states or tokens. This leads to difficulties with the current Playwright implementation. Scrapers are primarily needed when lightweight APIs aren't available or practical, and unfortunately in the modern web heavy pages of multiple megabytes of data can be included in each response.

If you're getting HTTP responses every two seconds, you can easily amass over 1GB of additional memory leakage within an hour if all of those responses are being stored. In my case, after 43 minutes my playwright node process crashed at 1.9GB of total memory size, with the error FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory. Increasing the memory limit will only delay the issue. Being able to prevent a context from cacheing all responses would be ideal.

[gigitalz]

I have a case where I make several requests per minute, it leaks memory all over the place. I've tried closing pages, closing contexts, closing browsers, catching all the onClose events and reclosing everything all together, even closing the main playwright instance all the time, nothing works, it still leaks memory and at some point the GC thrashes completely with the CPU setting on fire.

Please provide and/or document a way to fully release all resources so that we can clear it out every now and then. Please don't just focus on testing, it's full of other use cases where playwright is needed to run long long times.

[Xzeffort]

when I run job for a long time, the memory used too much.

total Memory 4GB

[gigitalz]

Is there a way to properly dispose the whole thing without terminating the process launching it?

[rigwild]

I have a case where I make several requests per minute, it leaks memory all over the place. I've tried closing pages, closing contexts, closing browsers, catching all the onClose events and reclosing everything all together, even closing the main playwright instance all the time, nothing works, it still leaks memory and at some point the GC thrashes completely with the CPU setting on fire.

Please provide and/or document a way to fully release all resources so that we can clear it out every now and then. Please don't just focus on testing, it's full of other use cases where playwright is needed to run long long times.

I have the exact same issue. Use case: scrapping/botting.

Here is a ultra-simple repro:

import { chromium } from 'playwright'

const setup = async () => {
  const browser = await chromium.launch({ headless: false })
  let page = await browser.newPage()

  let j = 0
  while (true) {
    for (let i = 0; i < 20; i++, j++) {
      console.log(j, i)
      await page.goto('https://v3.vuejs.org/guide/introduction.html#declarative-rendering')
    }
    console.log('Trying to create a new context, does not fix the leak!')
    await page.close()
    page = await browser.newPage()
  }
}

setup()

Both this and #8775 are probably duplicates. It shows that there is an issue with playwright itself and not with chromium or webkit.

For readers having the same issue, as a temporary workaround you might use something like PM2 with the memory limit config. It will restart the process when the limit is reached.
This is far from ideal though.

[mrmiroslav]

Hey guys - any ETA on when is this going to be fixed?

[LuizFelipeNeves]

Something that can help if it hasn't already been done is optimizing requests, ignoring unnecessary things like images.

[LDubya]

@LuizFelipeNeves images can be very important, depending on what you are doing. Automated browsers have lots of different use cases. The project that I used playwright for needed to take screenshots of webpages, images and all. The problem that Playwright is having isn't a request issue, but a memory leak issue. Reducing the size of the requests only makes it so that it takes longer for the memory overflow to occur, it doesn't fix the underlying problem. I think we can all agree that we should be able to instruct playwright to not make an infinite cache of request response data between page loads, while caching the things that are designed to persist between page loads like cookies and local storage. Basically, a setting to behave more like a browser. So much of the issue we're facing with playwright was solved by the early browser developers dating back to Netscape's cookies in 1994; there's no need to reinvent the wheel here.

[vonkoff]

still same issue

Using the code I use in my project?

[tzbo]

I think it's a stupid design in playwright. Any programmer will hate the memory problem and that is not in his control.

[LeMoussel]

@vonkoff It's not Windows compliant.

On Windows machine, to kill a Node.js server, and you don't have any other Node processes running, you can tell your machine to kill all processes named node.exe. That would look like this:

taskkill /im node.exe

And if the processes still persist, you can force the processes to terminate by adding the /f flag:

taskkill /f /im node.exe

[vinismarques]

Besides not being a proper solution, I would just like to make it clear that restarting everything is just not feasible is some cases, like when you are working on a infinite scroll page. If you restart, you are doomed.

Thought I would reiterate on this just so that no one thinks there is a good way to solve this issue. The bug should be fixed.

[dgtlmoon]

I think it's a stupid design in playwright. Any programmer will hate the memory problem and that is not in his control.

@tzbo tough comment, why dont you make something better?

[gigitalz]

@tzbo tough comment, why dont you make something better?

This is the usual dump f. take people come with when they have zero priority to solve a ticket.

[tzbo]

It's tough and it's a long time ticket. There is no better solution in two years to solve memory problem. Do you think it's a good design? Maybe the original intention is to simplify usage. Of courese it's more simple than puppeteer in some senario. But I think at least it should keep some release memory funtions which mean I call these funtions. I promise I will not use previouse response and ....
context.close? no, it can not release memory. I used it at previous version. Another, I don't want to close it.So I migrate to pupppeter.
I wish playwright will be better soon. It supports more browsers.

[ldexterldesign]

👋 @gigitalz,

dgtlmoon isn't well socialised

Ignoring is the only language he speaks

Regards

🥩

[lanyuer]

This is also a very confusing issue for us. As long as we keep opening new pages within the loop, the memory keeps increasing continuously. We have encountered several OOM errors recently.

[pavelfeldman]

We have all the feedback we need for this issue and it is currently pending due to prioritization. I'll disable the comments since they no longer add actionable details to the issue.

[pavelfeldman]

Unbounded heap growth should be mitigated by ffd20f4. The heap will still saturate to a certain size (1K handles per object type, ~50-100Mb on average), but will stop growing afterwards.

[dgtlmoon]

Apart from making demands to people I dont know like some previous commenters, I would like to thank Pavel for adding that heap stack test to npm, that's a super cool idea. And generally thank the maintainers for their incredible work here, it's a highly complex project!

[gauravkhuraana]

I face this issue in Azure Dev Ops agent pipeline. Locally all test run fine.. we have 200+ test and 5-6 run in parallel at a time.. after 8 minutes of running build it fails with

<--- Last few GCs --->
[12856:000002C6125D7660] 111 ms: Scavenge 7.9 (9.1) -> 7.6 (9.8) MB, 0.9 / 0.0 ms (average mu = 1.000, current mu = 1.000) allocation failure;
[12856:000002C6125D7660] 142 ms: Scavenge 8.4 (9.8) -> 8.0 (10.1) MB, 0.8 / 0.0 ms (average mu = 1.000, current mu = 1.000) allocation failure;
[12856:000002C6125D7660] 192 ms: Scavenge 8.7 (10.1) -> 8.2 (10.1) MB, 35.7 / 0.0 ms (average mu = 1.000, current mu = 1.000) allocation failure;
<--- JS stacktrace --->
FATAL ERROR: NewSpace::Rebalance Allocation failed - JavaScript heap out of memory
1: 00007FF621C4194F
2: 00007FF621BC6026
3: 00007FF621BC7D10
4: 00007FF6226721F4
5: 00007FF62265D582

[ludmilanesvitiy]

After update from 1.29.1 to the latest 1.40.0, started to receive the error:
page.goto: The object has been collected to prevent unbounded heap growth.
And yes, I know that I'm not using page.close() to close the window after each test, because it's necessary to speedup tests. With closing page after each test, the error about memory usage disappeared, but appearing another problems, which I do not need to solve.

So, for now decided to downgrade to prev version.
Any fix to the lib will be appreciated. Let me know if it's done in some new release. Thanks.

PS. I have about 800 scenarios, each of them has about 15 separate steps. In average, on the 300-350th test they become to fail.

[mxschmitt]

We need unfortunately a reproduction case in order to debug issues like that. A small repository would be ideal. I'm going to lock this for now, so others can re-file and we can work on the missing scenarios, thanks!