Do different crawling projects share a browser pool?

[kanxue660]

Hi All,

If I have written two crawling projects that are started simultaneously, do they share a browser pool, or does each crawling project initiate its own browser pool?

[janbuchar]

Hello, every crawling project has its own browser pool, or more precisely, every node process used to start a crawl has its own browser pool. Sharing them would be difficult with the current Crawlee.

[B4nan]

To be even more precise, every crawler instance has its own browser pool:

crawlee/packages/browser-crawler/src/internals/browser-crawler.ts

Line 449 in f3927e6

this.browserPool = new BrowserPool<InternalBrowserPoolOptions>({

[kanxue660]

Thank you for your answer in advance. So if this happens, what is the point of a browser pool for each website as a project? I don’t pursue speed, but I will crawl many different websites and have different page structures. How to share the browser pool? Otherwise, each project will have to start and close the browser, worrying about occupying system resources

[barjin]

The current purpose of BrowserPool is to handle browser management during the crawler run and provide a unified interface for opening / closing pages in the managed browsers + handle fingerprint injection and proxy setup.

There indeed might be a performance hit from not reusing the managed browsers across multiple concurrent crawls. Unfortunately, right now, there is no way of instantiating the BrowserPool separately and passing it to the crawler instance. While there might be actual technical reasons for this (e.g. the way that proxies currently bind to running browsers), this is IMO rather a design oversight.

Currently, your best bets are:

1. Run the projects separately and accept the performance hit (it might not be too severe, especially if you're not running a separate crawler for 1-2 URLs)
2. Combine the codebases of your projects and run all of them in a single crawler instance. This will clutter your codebase, though. You can mitigate that by splitting the behavior branches using a Router instance.

to other maintainers: I added this to the internal Crawlee v4 proposed features Notion document.