Working with 1.18 with 2 playwrights when the 2 object gets closed it throws HeapSpace #827
Comments
|
You are running out of memory, looks like the machine doesn't have enough memory to run multiple playwrights on it. If you still think this is a bug in the playwright please provide a repro that we could run locally. |
|
I am running with -Xms3g -Xmx3g when the shutdown is call i am using like 800 mb later the Out of memory exception is thrown when i call shutdown on the second playwright i will test this the same scenario with more memory. |
|
Running with -Xms20g -Xmx20g with connecting to a wsEndPoint Browserless in this case. When trying to close Page object it throws 20 GB is a lot of memory. |
|
Can you share a repro that we could run locally? |
I will try that today mate. :) |
|
When running the process with 20G of ram memory the 2 page object it hangs the process as you can see here it takes more than 10 minutes hang i have try even longer 2 or 3 hours and continues hanged. |
|
If i run with flag -XX:+PrintGCDetails i can see next and immediately after the call to the 2 Page object shutdown method the message
Using -XX:+UseG1GC the problem continues closes the first Page and hangs and first browser Object shutdown
|
|
Strangely inside IntellijIdea applying System.exit(0) with 10 playwrights with -Xmx:5g works like a charm. I will dive more on this.
|
|
I can reproduce with this sample of course the running code is harder and better designed but i think will reproduce it.. import com.microsoft.playwright.Browser;
import com.microsoft.playwright.BrowserType;
import com.microsoft.playwright.Page;
import com.microsoft.playwright.Playwright;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class StreamClosed{
private final List<DataPerThread>dataPerThreads = new ArrayList<>();
private final CountDownLatch latch = new CountDownLatch(5);
private static final double MB_VALUE = 1024 * 1024;
private final List<String> urls = new ArrayList<>();//will be using only one in all 5 threads for the sake of simplicity
interface ShutdownEvent {
void shutdown();
}
public static void main(String[] args) throws Exception{
final StreamClosed clazz = new StreamClosed();
final Runnable hook = ()->clazz.dataPerThreads.forEach(DataPerThread::shutdown);//CTRL + C WAS PRESSED
Runtime.getRuntime().addShutdownHook(new Thread(hook));
ExecutorService service = Executors.newFixedThreadPool(5);
for(final String url:clazz.urls){
final ThreadCaller thread = clazz.new ThreadCaller("http://myexternalip.com/raw");
System.out.println("Send work with thread: "+thread.getName());
service.execute(thread);
}
clazz.waitForChildThreadsToComplete();
service.shutdown();
service.shutdownNow();
System.out.println("Ends....");
try {
Thread.sleep(240_000);
clazz.dataPerThreads.forEach(DataPerThread::shutdown);//IF HERE NOT CTRL+C WAS PRESSED
}catch (final Exception e){
e.printStackTrace();
}
}
protected void waitForChildThreadsToComplete() {
try {
latch.await();
System.out.println("All child threads have completed.");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private final class ThreadCaller extends Thread{
private final String url;
public ThreadCaller(final String url) {
this.url = url;
}
@Override
public void run() {
new DataPerThread(url).run();
}
}
private final class DataPerThread implements ShutdownEvent,Runnable {
protected final Logger LOGGER = LoggerFactory.getLogger(DataPerThread.class);
private final Playwright playwright;
private final String url;
private final BrowserType browserType;
private Browser browser;
private Page page;
private final int playwrightHashCode;
public DataPerThread(final String url) {
LOGGER.info("Building DataPerThreadObject for Thread:[{}]",Thread.currentThread().getName());
LOGGER.info("Creating playwright with thread [{}].",Thread.currentThread().getName());
final long startPlayWright = System.currentTimeMillis();
this.url = url;
this.playwright = Playwright.create();
dataPerThreads.add(this);
this.playwrightHashCode = System.identityHashCode(this.playwright);
LOGGER.info("Working with Playwright:[{}] took:[{}] ms.",this.playwrightHashCode,(System.currentTimeMillis()-startPlayWright));
LOGGER.debug("Creating browserType [{}]",this.playwrightHashCode);
final long creatingBrowsersInfo = System.currentTimeMillis();
this.browserType = playwright.chromium();
LOGGER.debug("Created browserType [{}] with Playwright:[{}]",System.identityHashCode(this.browserType),this.playwrightHashCode);
fillPlayWrightProxy();
LOGGER.info("Working with browsers objects :[{}] with Playwright [{}] took:[{}] ms.",System.identityHashCode(browserType),playwrightHashCode,(System.currentTimeMillis()-creatingBrowsersInfo));
LOGGER.info("Builded DataPerThreadObject [{}] for Thread:[{}]",System.identityHashCode(this),Thread.currentThread().getName());
}
@Override
public void run() {
System.out.println("Connecting to: "+url+" with Thread:[{}]"+Thread.currentThread().getName());
try {
System.out.println("Requesting: "+url+" with page: "+(System.identityHashCode(page)));
System.out.println(page.navigate(url).body().length);
}catch(final Exception e){
System.out.println(e.getLocalizedMessage());
}
latch.countDown();
System.out.println("Left: "+latch.getCount());
}
private void fillPlayWrightProxy(){
LOGGER.info("BrowserType: [{}]",System.identityHashCode(browserType));
this.browser = browserType.connect("wss://chrome.browserless.io/playwright?token="+token);
LOGGER.debug("Created pageProvider [{}] with BrowserType:[{}]",System.identityHashCode(this.browser),System.identityHashCode(this.browserType));
this.page = newPage();
}
private Page newPage(){
LOGGER.debug("Trying to open page with Playwright Object:[{}]",playwrightHashCode);
final long start = System.currentTimeMillis();
final Page localPage = this.browser.newPage();
localPage.onClose(this::onClosePage);
final long end = System.currentTimeMillis();
LOGGER.debug("Building page [{}] for thread [{}] with playwright: [{}] took {} ms.",System.identityHashCode(localPage),Thread.currentThread().getName(),playwrightHashCode,(end - start));
return localPage;
}
private void onClosePage(final Page page){
LOGGER.debug("Page [{}] will be closed shutDownProcess started:[{}].",System.identityHashCode(page),false);
}
@Override
public void shutdown() {
final String threadName = Thread.currentThread().getName();
final long hashcode = System.identityHashCode(this);
LOGGER.info("Shutdown process start at [{}] with thread:[{}].",hashcode,threadName);
final boolean pageIsClosed = page.isClosed();
final boolean browserIsConnected = browser.isConnected();
LOGGER.debug("Page [{}] is closed [{}] browser is connected:[{}] browserContext:[{}].",System.identityHashCode(page),pageIsClosed,browserIsConnected,System.identityHashCode(page.context()));
if(!pageIsClosed && browserIsConnected){
close(page);
}
if(browserIsConnected){
close(browser);
}
close(playwright);
LOGGER.info("Shutdown process end at [{}] with thread:[{}].",hashcode,threadName);
}
private double getFreeMemory(){
return Runtime.getRuntime().freeMemory() / MB_VALUE;
}
private DataPerThread close(final AutoCloseable closeable){
if(closeable == null){
return this;
}
final String className = closeable.getClass().getSimpleName();
final int hashcode = closeable.hashCode();
try{
LOGGER.info("Closing [{}] object:[{}] with free memory:[{}] MB.",className,hashcode,getFreeMemory());
final long startTime = System.currentTimeMillis();
closeable.close();
final long endTime = System.currentTimeMillis();
LOGGER.info("Closed [{}] object:[{}]. Free---->{} [MB] took:[{}] ms.",className,hashcode,getFreeMemory(),(endTime-startTime));
}catch(final Exception e){
LOGGER.warn("Exception [{}]Closing [{}] object:[{}].",e.getLocalizedMessage(),className,hashcode);
}
return this;
}
}
}With 3GB and pressing [Ctrl+C] in sleep it throws With 35 GB it hangs forever after [Ctrl+C].
|
|
Does it reproduce with local browsers instead of browserless ? |
I will let you know tomorrow :) |
|
In the previous snippet i was using browserless not using Browserless i mean instead of using this.browser = browserType.connect("wss://chrome.browserless.io/playwright?token="+token); Using this.browser = browserType.launch(); With 3GB of ram it closes Page + Browser object but it throws heap closing the 1 playwright object.
Trying to understand the dump i see something like this.
With 35 GB it hangs closing the page of the second thread.
Please note that when the shutdown process started is the Thread-0 is the thread i think that will kill Java process. As you can see in the code i put a .sleep this is mainly for see what happens when a Ctrl+C is call or if not how behave the code when a the regular flow is not interrupted. If not CTRL+C is not pressed and the code flows normally even with 3GB of ram it closes Smoothly. As you can see here the thread is main instead of Thread-0
It seems it have what to do which Thread closes the objects :( After all the work is done and calling System.exit(0); even on a 1GB ram it works smoothly even with [Thread-1]
It seems that playwright is suffering when Ctrl+C is call/pressed or when AWS shutdowns the machine killing the Java process |
|
Try running it with |
Are this system properties o maven commands? we are using gradle.. |
These are environment variables. |
I will do this mate and i will let you my kid is in the hospital :( |
|
With 3GB ram OutOfMemoryException |
|
With Environment variables set. |
|
We fixed a related bug in 1.22, could you try to reproduce it there? I'm closing it in the mean time, since its most likely fixed. Please file a new issue if you still encounter such issues with a new repro, thanks, appreciate the effort! |
|
Hey mate with the latest Playwright version it works. 1.25 :) |
Here is the info
https://stackoverflow.com/questions/71242763/java-playwright-throws-java-lang-outofmemoryerror-java-heap-space-closing-a-pla?noredirect=1
And when we connect to a wsEndPoint the problem arises when we try to close Page and Browser objects but not with Playwright object as you can see here.
The text was updated successfully, but these errors were encountered: