Proxy rotation does not work with CamouFox

[galaczi]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/playwright (PlaywrightCrawler)

Issue description

Use something like:

proxyConfiguration: new ProxyConfiguration({
    newUrlFunction: (sessionId) => {
      const proxyUrl = `http://brd-customer-...-zone-...-session-${sessionId}:mypass@brd.superproxy.io:33335`;
      console.log('proxyUrl', proxyUrl);
      return proxyUrl;
    },
  }),

Unique sessionIds are correctly logged, but the browser uses the very first sessionId for all requests regardless of sessionId.

Package version

camoufox-js@0.3.5, crawlee@3.13.2, playwright@1.52.0

Node.js version

v20.13.1

Operating system

Ubuntu

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

We dont rotate browsers based on sessions by default, since it would mean opening a lot of browser contexts, which would eat up your memory. You can opt in to that behaviour via browserPerProxy option, see #2418 for more info.

[galaczi]

Hi, just to be sure I understand. You are saying regardless of what proxy url is provided, crawlee will use one proxy, unless browserPerProxy is set to true? I have also tried with a Playwright (chromium) project and proxies are rotated as I expected.

To reiterte, with Camoufox only one proxy is used, even though the newUrlFunction creates a new proxy url based on the new sessionId. With Chromium the proxies are rotated by sessionId, so they work differently?

[B4nan]

With Chromium the proxies are rotated by sessionId

I doubt it, what I described above applies to all browser crawlers, it's not just about camoufox. To have a different proxy with a browser, you need a new browser context, and we do that only after a browser is "retired", which happens after some amount of pages are processed with it (specifically, we have retireBrowserAfterPageCount and maxOpenPagesPerBrowser options in the browser pool). Alternative to the browserPerProxy option is using useIncognitoPages, which opens anonymous contexts for every page - the downside with that approach is you loose resource caching (js/css), which can hurt performance quite badly.

Note that when you use camoufox, you use the exact same code paths as with chromium, the only difference is the browser handle controlled by playwright. So not just our code, but also the whole playwright code, all of that is exactly the same.

Hard to say more, please provide a complete reproduction so it's clear what you are doing.