You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am using the workerpool library to manage multiple services in a Node.js application, specifically for crawling tasks using Playwright. However, I've encountered an issue with Playwright related to memory leaks. This seems to be a common problem among developers using Playwright, and the suggested workaround involves restarting the Playwright process to free up memory.
Issue
In my application, each service is a separate worker within workerpool. One of these services, a crawler, is responsible for handling thousands of URLs. Due to the memory leak in Playwright, I need a way to programmatically restart this specific service (crawler) within workerpool. The service is stateless and does not process any data persistently, so it should be feasible to restart it without losing important information.
Current Implementation
Here is a simplified version of how the services are structured:
// Main fileimportpathfrom'path';import{fileURLToPath}from'url';import{pool}from'workerpool';import{runApiServer}from'~/api/api.js';const__filename=fileURLToPath(import.meta.url);const__dirname=path.dirname(__filename);constcomputeWorkersCount=(name: AppWorker): [min: number,max: number]=>{return[Number(CONFIG[(name+'_WORKERS_MIN')asAppWorkersMin]),Number(CONFIG[(name+'_WORKERS_MAX')asAppWorkersMax]),]}constmain=async()=>{// Database initializationsconstoneDB=createOneDB();constanotherDB=createAnotherDB();runApiServer({ oneDB , anotherDB });constservices=[[computeWorkersCount('FIRST'),'./services/one'],[computeWorkersCount('SECOND'),'./services/second'],[computeWorkersCount('THIRD_LEAK_MEMORY_PROBLEM'),'./services/third'],];for(const[[min,max],servicePath]ofservices){pool(path.join(__dirname,servicePath),{minWorkers: min,maxWorkers: max,}).exec('main',null).catch(console.error);}};main();// Example of a service workerimport{worker}from'workerpool';worker({main: ()=>main({oneDB: createOneDB(),anotherDB: createAnotherDB(),}),});
Request
I am seeking guidance or a feature within workerpool that would allow me to restart a specific service (especially the crawler service using Playwright) to handle the memory leak issue. This would involve terminating and then reinitializing the service's process. Any suggestions or solutions for this scenario would be greatly appreciated.
The text was updated successfully, but these errors were encountered:
@josdejong I would like to make that logic, that worker process logic inside it (or best repeat it N times), and after that it will be terminated and re-created (with reset N counter) automatically.
I think what you can do is create a little wrapper function around your workerpool that:
creates a workerpool instance
keeps track on the number of executed tasks (divided by the number of workers to get your N tasks per worker)
once the max executed tasks is reached, gracefully shutdown the pool and create a new one
There is no support for terminating a single worker, but this would terminate all of them and re-create them once in a while to solve the memory leaks issue.
Description
I am using the
workerpool
library to manage multiple services in a Node.js application, specifically for crawling tasks using Playwright. However, I've encountered an issue with Playwright related to memory leaks. This seems to be a common problem among developers using Playwright, and the suggested workaround involves restarting the Playwright process to free up memory.Issue
In my application, each service is a separate worker within
workerpool
. One of these services, a crawler, is responsible for handling thousands of URLs. Due to the memory leak in Playwright, I need a way to programmatically restart this specific service (crawler) withinworkerpool
. The service is stateless and does not process any data persistently, so it should be feasible to restart it without losing important information.Current Implementation
Here is a simplified version of how the services are structured:
Request
I am seeking guidance or a feature within
workerpool
that would allow me to restart a specific service (especially the crawler service using Playwright) to handle the memory leak issue. This would involve terminating and then reinitializing the service's process. Any suggestions or solutions for this scenario would be greatly appreciated.The text was updated successfully, but these errors were encountered: