Recommended Crawlee setup for multiple isolated Firefox sessions sharing one IP? Hey Crawlee team,

[MShneur]

I'm experimenting with Crawlee on an Oracle Cloud Linux server and could use some advice on the recommended setup.

What we're trying to achieve:

Imagine an airport Wi-Fi network: multiple devices, all sharing one public IP address.

We want to test a similar environment using Crawlee:

• One VPN connection and one shared public IP.
• Multiple independent browser workers running simultaneously.
• Each worker maintains its own cookies, storage, browser context, and consistent device fingerprint.
• Each worker can maintain multiple tabs without mixing sessions.
• Everything runs locally on Oracle, without paid scraping APIs.

Our current setup:

We're using Crawlee JavaScript 3.18.2, Playwright 1.64.1, Node 20, and Firefox.

We've successfully launched two isolated workers with separate cookies on a local test page. However, we're unsure whether we're configuring Crawlee's BrowserPool, SessionPool, and built-in fingerprint features correctly.

A few questions:

1. What's the recommended native architecture for multiple independent Firefox browser sessions sharing one IP?
2. Can Crawlee automatically manage separate, stable browser fingerprints and session storage for each worker?
3. Which browser-pool and session features should we enable together, and are there known Firefox/Linux compatibility limitations?
4. Is there an existing example or reference project demonstrating this setup?

We're testing browser-session isolation and concurrency, not looking to circumvent website security restrictions.

Any recommended documentation, examples, or configuration guidance would be appreciated.

Thanks!

[slophisticated]

For the multi-tab requirement, I would make one long-lived Playwright BrowserContext the boundary of each logical worker, then open that worker's tabs with context.newPage(). Multiple contexts can share one Firefox browser process while keeping browser storage isolated (Playwright isolation guide). Sharing an outbound IP is separate from that storage boundary; creating contexts does not create new IPs.

A few distinctions in Crawlee 3.18.2 matter here:

• useIncognitoPages: true isolates each pool-created page, rather than grouping several independently scheduled pages into one worker context. The 3.18.2 controller calls Playwright's browser.newPage() in that mode. Conversely, the non-incognito launch path uses a persistent context, so pages on the same controller can share state. Neither setting alone defines your logical-worker-to-context ownership.
• A SessionPool Session is not a BrowserContext. Its persisted state includes a cookie jar and session metadata/user data, not an automatic snapshot of localStorage, IndexedDB or every tab's sessionStorage.
• Generated fingerprints are not automatically unique, durable worker identities. The 3.18.2 fingerprint hook uses launchContext.session?.id ?? launchContext.proxyUrl as its cache key and attaches the fingerprint to the context. When no session id is supplied, contexts using the same proxy URL can therefore reuse the cached fingerprint. For isolation tests, I would first keep each worker's explicit browser/context settings fixed and test storage separation without randomization.

For restart persistence, use a separate state file per worker, or a separate userDataDir per persistent worker context. Playwright does not allow concurrent browser instances to use the same profile directory. Also, sessionStorage is tab-scoped, and Playwright's ordinary storage-state persistence does not save it.

For a reference starting point, the official multiple-roles example demonstrates explicit context ownership. I'd extend each context with additional tabs before adding crawler scheduling. If you need Crawlee's scheduler, the missing part to design explicitly is worker affinity and what happens when a session/browser is retired; simply increasing concurrency or pool size does not specify that policy.

Do your workers need to survive browser restarts, or only keep their tabs isolated during one run? And could you share your current launchContext, browserPoolOptions and sessionPoolOptions? Those would help distinguish a configuration change from a need for custom lifecycle management.

AI-assisted, source-checked against Crawlee v3.18.2 and the linked Playwright documentation; not runtime-tested on your Oracle Linux/Firefox setup.

[MShneur]

Thanks, this helped clarify the issue.

We're using Crawlee 3.18.2 with Playwright/Firefox on Oracle Linux.

We need isolation during a single run, not persistence across restarts.

I tested your proposed structure locally: one Firefox process, two explicit contexts, two tabs per context. All four navigated successfully, and cookies remained isolated.

Our previous Crawlee configuration used "useSessionPool: true", "persistCookiesPerSession: true", "sessionPoolOptions: { maxPoolSize: 2 }", and "launchContext.useIncognitoPages: true". We confirmed that SessionPool alone did not guarantee the worker-to-context relationship we wanted.

One follow-up: What's the cleanest supported way in Crawlee 3.18.2 to attach scheduled requests to a specific, long-lived BrowserContext, while allowing multiple tabs in that context?

Would you recommend BrowserPool hooks/custom controllers, separate crawler instances, or managing the worker contexts directly and using Crawlee only for scheduling?

Our goal is reliable category pagination and accurate product collection, with a measurable throughput improvement over the original collector.

[slophisticated]

For your exact requirement, I'd keep the contexts you have already tested and use BasicCrawler for request scheduling, with your own Playwright context/page lifecycle. That is an application design using public APIs, not a built-in worker-affinity option.

Why: in 3.18.2 BrowserCrawler, each request opens a page through browserPool.newPage(). The pool implementation selects a controller with capacity before running page-creation hooks; the page id identifies a page, not a requested BrowserContext. I wouldn't use a hook to retrofit ownership, or depend on overriding private controller-selection internals.

A concrete structure is:

• Launch Firefox once; create your two contexts before the crawl. Keep a map from an application workerId to its context.
• Put workerId in request userData, and carry it forward when enqueueing pagination/product requests. In the BasicCrawler requestHandler, select that context, open a page with context.newPage(), navigate/extract, then close the page in finally. Close the contexts/browser when the run finishes. BasicCrawler doesn't supply or manage those pages for you.
• Bound both total concurrency and tabs per context. Start with serial category pagination within each worker; parallelize independent product pages once correctness is established. Cookies/localStorage are shared within a context, so parallel tabs can still affect each other's application state.
• Keep retries on the same worker identity and make output deduplication explicit. If the same URL must run under both identities, use distinct request uniqueKey values; otherwise normal queue deduplication can collapse them. Define what happens if a context/browser dies.

Separate crawler instances don't themselves guarantee one retained context; each still has its own pool lifecycle. For throughput, compare the same category set and product-ID coverage, plus wall time, retries and missing/duplicate records. More tabs alone isn't evidence of improvement.

AI-assisted, source-checked against 3.18.2; I haven't runtime-tested this scheduler integration on Oracle/Firefox.

[MShneur]

Thanks again. We implemented your BasicCrawler approach on Oracle using Crawlee 3.18.2.

Results: Two persistent Firefox contexts, worker-owned pagination, and six queued requests all passed without errors or context crossover.

We also isolated the retailer issue: plain Firefox without Crawlee encounters the same catalog-access refusal. We're treating that separately.

Before we close out the Crawlee integration, I have two questions:

1. What's the recommended recovery pattern when a worker-owned BrowserContext crashes while requests remain queued?
2. How should we verify that retries and queue completion never silently lose requests, especially when ownership must remain fixed?

Our goal is reliable, sustained collection with verified saved records, not simply increasing tab counts.

Appreciate your guidance.

[slophisticated]

Thanks for testing that on Oracle and separating the retailer refusal. For this custom BasicCrawler setup, I would keep recovery in your worker-context manager and let Crawlee manage request retries.

1. Recover the context without changing the logical owner

• Keep request.userData.workerId and the owner-qualified uniqueKey unchanged on retry. A replacement context belongs to that same worker; do not route its pending requests through another worker's context.
• Give each worker a context generation and one shared recovery promise/lock. On context close, mark that generation unavailable. If the shared browser disconnects, invalidate all contexts belonging to it and serialize browser relaunch too. Requests for a recovering worker must wait for its replacement, with a bounded timeout. Only publish a new generation after initialization and an isolation/health check succeed. Compare generations in event handlers so a late close event from an old context cannot invalidate its replacement.
• Let interrupted navigation/extraction throw from requestHandler. Do not catch it and return successfully, or manually mark/reclaim that request inside the handler. Set a finite maxRequestRetries; Crawlee will reclaim ordinary failures, preserving the request's owner data. Recovery itself also needs bounded attempts/backoff. If you need to suspend an unavailable worker indefinitely, use an explicit pause/resume design, such as separate worker queues, rather than unlimited retries.
• Reapply the worker's own context options and authentication initialization. A newly created context cannot recover unsaved in-memory state. If you need state continuity after a crash, checkpoint each worker while healthy and restore its last valid checkpoint; storageState is not a complete browser/tab snapshot. For actual launchPersistentContext profiles, keep a separate profile directory per worker and never open the same directory concurrently. A single crashed page can instead be replaced without retiring a healthy context.

2. Make completion an independently checked result

Treat request execution as replayable, and make saved output idempotent:

• Before scheduling, durably record each intended (crawlRunId, workerId, requestKey) in a manifest. For discovered pagination/product requests, persist their intent and await enqueueing before the parent handler returns. Recover incomplete enqueue operations from that manifest.
• Await validation and a committed output write before returning from requestHandler. Use a stable record key/upsert or transaction, so a crash after the write but before queue acknowledgement does not duplicate records. A successful request should have a completion receipt describing its verified records, including an explicitly validated empty result where appropriate.
• Record exhausted requests in failedRequestHandler, with owner, key, retry count and error. Important in 3.18.2: Crawlee calls markRequestHandled() before failedRequestHandler. Thus queue.isFinished() can be true with failures, and a crash/failure in that callback can leave no failure receipt. Reconcile against your independent manifest; do not rely on that callback alone as the loss detector.
• For process restarts, use the same durable named queue/storage and disable startup purging (purgeOnStart: false); an ephemeral queue cannot provide restart recovery. Preserve the manifest and output store separately as well.

I would gate a successful run on set reconciliation, per worker: every intended request has exactly one verified success receipt or an explicit terminal-failure receipt, no unexplained keys remain, and saved record keys/counts match the success receipts. Report terminal failures separately rather than calling the whole collection successful.

For your next test, inject context closure during navigation, shared-browser loss, a sink failure, failure just after an output commit, exhausted retries, and process termination/restart. Assert unchanged ownership on every attempt and reconcile the actual saved records afterward. This checks much more than tab count or queue drain.

I checked the 3.18.2 retry/failure implementation and ran a small scheduler/sink fault-injection probe with @crawlee/basic@3.18.2 on Windows/Node 24.20.0: six distinct owner-qualified requests, 14 attempts, four idempotently saved results, two explicit exhausted failures, unchanged owners, and a finished queue. Both failure callbacks observed an already-handled request. This used simulated context loss and in-memory storage; it does not validate Firefox recovery, durable writes or process restart on Oracle. Playwright documents the browser disconnect event and context storage state.

[janbuchar]

CC @Mantisus — perhaps you might have an idea how to handle this?

[Mantisus]

I have a couple of ideas. Neither requires building your own browser context lifecycle, but the second one relies on a few hacks and internal implementation details. I'm not sure yet how much better this can be solved in v4.

Crawlee already binds a browser to a session. With useIncognitoPages: false (the default), each browser runs a single persistent context. At launch it gets one Session from the pool, and every page in that browser uses it. The fingerprint is cached per session id, so it follows the session as well. So navigation within one browser always runs under one session, and that browser's tabs share cookies and storage.

The problem to solve is the other direction. One session can end up in several browsers, because a newly launched browser gets a random session from the pool. And a request can go to any browser with a free tab.

The simplest option: one crawler per worker

Give each worker its own crawler, with its own request queue and a session pool of exactly one session. Once the pool is full, getSession() returns a session from the pool. So every browser the crawler launches, including after a crash, gets the same session back, and with it the same fingerprint and cookies. Pagination enqueued by a worker goes into its own queue, so it never reaches another worker.

import { PlaywrightCrawler, RequestQueue } from 'crawlee';
import { firefox } from 'playwright';

/**
 * One crawler per worker: its own RequestQueue, a SessionPool with a single Session,
 * and one Firefox browser whose tabs share one persistent context.
 */
export async function createWorkerCrawler(worker, { tabs, requestHandler, ...options }) {
    return new PlaywrightCrawler({
        ...options,
        // A separate queue per worker, so pagination and retries stay with the worker.
        requestQueue: await RequestQueue.open(`worker-${worker}`),
        launchContext: { launcher: firefox },
        useSessionPool: true,
        persistCookiesPerSession: true,
        sessionPoolOptions: {
            // A single Session: every (re)launched browser of this crawler gets the same one back.
            maxPoolSize: 1,
            persistStateKey: `SESSION_POOL_${worker}`,
            // Defaults would rotate the identity after 50 pages, 50 minutes or 3 errors.
            sessionOptions: { maxUsageCount: 1_000_000, maxAgeSecs: 86_400, maxErrorScore: 50 },
        },
        browserPoolOptions: {
            maxOpenPagesPerBrowser: tabs,
            // Defaults would close the browser after 100 pages or 10 s without a new page.
            retireBrowserAfterPageCount: 1_000_000,
            retireInactiveBrowserAfterSecs: 3_600,
        },
        maxConcurrency: tabs,
        async requestHandler(context) {
            await requestHandler(context);
            // Crawlee saves cookies right after navigation; also keep cookies set by JS during the handler.
            const { session, browserController, page, request } = context;
            session.setCookies(await browserController.getCookies(page), request.loadedUrl);
        },
    });
}

const startUrls = { w1: 'https://example.com/category/a', w2: 'https://example.com/category/b' };

await Promise.all(
    Object.entries(startUrls).map(async ([worker, url]) => {
        const crawler = await createWorkerCrawler(worker, {
            tabs: 3,
            async requestHandler({ enqueueLinks }) {
                // ... extract and save products ...
                await enqueueLinks({ selector: 'a.next' }); // stays in this worker's queue
            },
        });
        await crawler.run([url]);
    }),
);

Named queues are not purged on start (only the default storages are), so drop them between runs or run with CRAWLEE_PERSIST_STORAGE=false.

A single crawler, with preLaunchHooks

If you'd rather keep one crawler and one queue, both problems can be handled with preLaunchHooks:

1. Route requests by worker. With launchContext.browserPerProxy: true, BrowserPool picks the browser by its proxy URL, and ProxyConfiguration.newUrlFunction receives the request. So newUrlFunction can return a dummy URL built from request.userData.worker, which serves only as a routing key.
2. Don't actually use that proxy. BrowserPool records the requested proxy URL on the browser controller, but the browser's real proxy is read from launchContext.proxyUrl only after the preLaunchHooks have run. The hook clears it, and the browser connects directly over your shared IP.
3. Pin the worker's session. User preLaunchHooks run after the crawler's own hook, which picked the random session, and before the fingerprint hook. So the hook can swap in the worker's own session, and the fingerprint follows it. When the worker's browser is relaunched after a crash or rotation, it gets the same session back: the same fingerprint, and cookies re-applied from the session before the next navigation.

This option relies on internal behaviour of Crawlee 3.18.2, so pin the Crawlee version. Also, sendRequest from the crawling context would go through the dummy proxy URL, so don't use it with this option.

Draft implementation:

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
import { firefox } from 'playwright';

// The "proxy URL" is only a routing key; it is removed before the browser launches.
// `.invalid` is a reserved TLD, so the key can never resolve to a real proxy.
const workerKey = (worker) => `http://${worker}.worker.invalid`;
const workerFromKey = (proxyUrl) => new URL(proxyUrl).hostname.replace(/\.worker\.invalid$/, '');

/**
 * One Firefox process per worker, all of its tabs share one persistent context,
 * one Session (cookies) and one fingerprint. Every request is routed by `request.userData.worker`.
 */
export function createWorkerAffineCrawler({ workerCount, tabsPerWorker, requestHandler, ...options }) {
    const workerSessions = new Map(); // worker -> id of its current Session
    let generation = 0;

    const crawler = new PlaywrightCrawler({
        ...options,
        launchContext: { launcher: firefox, useIncognitoPages: false, browserPerProxy: true },
        proxyConfiguration: new ProxyConfiguration({
            newUrlFunction: (_sessionId, { request } = {}) => {
                const worker = request?.userData.worker;
                if (!worker) throw new Error(`Request ${request?.url} has no userData.worker`);
                return workerKey(worker);
            },
        }),
        useSessionPool: true,
        persistCookiesPerSession: true,
        sessionPoolOptions: {
            // Defaults would rotate the identity after 50 pages, 50 minutes or 3 errors.
            sessionOptions: { maxUsageCount: 1_000_000, maxAgeSecs: 86_400, maxErrorScore: 50 },
        },
        browserPoolOptions: {
            maxOpenPagesPerBrowser: tabsPerWorker,
            // Defaults would close a worker's browser after 100 pages or 10 s without a new page.
            retireBrowserAfterPageCount: 1_000_000,
            retireInactiveBrowserAfterSecs: 3_600,
            preLaunchHooks: [
                async (_pageId, launchContext) => {
                    const worker = workerFromKey(launchContext.proxyUrl);
                    // BrowserPool already recorded the key for browser selection; launch without a proxy.
                    launchContext.proxyUrl = undefined;

                    // Give the (re)launched browser the worker's Session instead of a random one,
                    // so the fingerprint (cached per Session id) and the cookies survive a relaunch.
                    const pool = crawler.sessionPool;
                    let session = workerSessions.has(worker)
                        ? await pool.getSession(workerSessions.get(worker))
                        : undefined;
                    if (!session) {
                        const id = `${worker}-${++generation}`;
                        await pool.addSession({ id });
                        session = await pool.getSession(id);
                        workerSessions.set(worker, id);
                    }
                    launchContext.extend({ session });
                },
            ],
        },
        maxConcurrency: workerCount * tabsPerWorker,
        async requestHandler(context) {
            await requestHandler(context);
            // Crawlee saves cookies right after navigation; also keep cookies set by JS during the handler.
            const { session, browserController, page, request } = context;
            session.setCookies(await browserController.getCookies(page), request.loadedUrl);
        },
    });

    return crawler;
}

const crawler = createWorkerAffineCrawler({
    workerCount: 2,
    tabsPerWorker: 3,
    async requestHandler({ request, enqueueLinks }) {
        const { worker } = request.userData;
        // ... extract and save products ...
        await enqueueLinks({ selector: 'a.next', userData: { worker } }); // pagination stays with the worker
    },
});

await crawler.run([
    { url: 'https://example.com/category/a', userData: { worker: 'w1' } },
    { url: 'https://example.com/category/b', userData: { worker: 'w2' } },
]);

I tested both options with Crawlee 3.18.2 and Firefox, including forced browser crashes.