session id is diffrent in `newUrlFunction` and `requestHandler` #2726

slow-groovin · 2024-10-28T13:08:14Z

slow-groovin
Oct 28, 2024

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/core

Issue description

The actual performance of IP Rotation and session management is confusing and not functioning properly:

please view the sample code, in short, the problem is:
newUrlFunction 's sesson id is diffrent to requestHandler 's session id of same request
session ids of different url in requestHandler are same, but the corespond proxys are different. (should the sessions of first five requests be same or not?)

newUrlFunction session_czoeqotOJe https://www.example.com?p=2 50801
newUrlFunction session_fEjJ4oqwit https://www.example.com?p=3 50806
newUrlFunction session_wNGcQewa9i https://www.example.com?p=4 50805
newUrlFunction session_gzL6MK5DHF https://www.example.com?p=1 50800
newUrlFunction session_J0cEZbF6di https://www.example.com?p=5 50805

preNavigationHooks about:blank https://www.example.com?p=2 session_ueHca5hDcR  50801
preNavigationHooks about:blank https://www.example.com?p=3 session_ueHca5hDcR  50806
preNavigationHooks about:blank https://www.example.com?p=4 session_ueHca5hDcR  50805
preNavigationHooks about:blank https://www.example.com?p=1 session_ueHca5hDcR  50800
preNavigationHooks about:blank https://www.example.com?p=5 session_ueHca5hDcR  50805

requestHandler https://www.example.com/?p=2 session_ueHca5hDcR 50801
requestHandler https://www.example.com/?p=1 session_ueHca5hDcR 50800
requestHandler https://www.example.com/?p=3 session_ueHca5hDcR 50806
requestHandler https://www.example.com/?p=4 session_ueHca5hDcR 50805
requestHandler https://www.example.com/?p=5 session_ueHca5hDcR 50805

Code sample

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';



const startUrls = [
    'https://www.example.com?p=1',
    'https://www.example.com?p=2',
    'https://www.example.com?p=3',
    'https://www.example.com?p=4',
    'https://www.example.com?p=5',

];


function createProxyConfigurationFromJson(){
	return new ProxyConfiguration({
		newUrlFunction: (sessionId,options)=>{
            const s=sessionId+''
			const port=s.charCodeAt(s.length-1)%10+50800
			console.log('newUrlFunction',sessionId,options?.request?.url,port)

			return `http://localhost:${port}`
		}
	})
}

const crawler = new PlaywrightCrawler({
	headless: true,
	maxRequestRetries: -1,
	navigationTimeoutSecs: 100,
	proxyConfiguration: createProxyConfigurationFromJson(),
    minConcurrency:5,
	maxConcurrency: 5,
	maxRequestsPerMinute: 500,
	useSessionPool: true,
	persistCookiesPerSession: true,

	preNavigationHooks:[
		async (context)=>{
			const {page,log,session,proxyInfo,request}=context
			console.log('preNavigationHooks',page.url(),request.url,session?.id,session?.getCookieString(request.url),proxyInfo?.port) //session id in here is same with requestHandler
		}
	],
	async requestHandler({page, proxyInfo, session}) {
        console.log('requestHandler',page.url(),session?.id,proxyInfo?.port)
    }
})
await crawler.run(startUrls);

Package version

3.11.5

Node.js version

20

Operating system

Win10

Apify platform

Tick me if you encountered this issue on the Apify platform

I have tested this on the `next` release

No response

Other context

No response

B4nan · 2024-10-28T15:49:46Z

B4nan
Oct 28, 2024
Maintainer

We don't rotate proxies on browsers like this, since it's not possible to change a proxy in an existing browser context. We do have an option to enable this behavior, but it will result in creating a new browser context for each request till you reach the session pool size, so you can easily exhaust your memory. So if you want to use that, be sure to limit the session pool size, it defaults to 100 which is way too many for this option.

https://crawlee.dev/api/browser-pool/interface/LaunchContextOptions#browserPerProxy (see the PR for mode details: #2418)

const crawler = new PuppeteerCrawler({
    // ...,
    launchContext: {
      browserPerProxy: true,
    }
});

Alternatively, you could use incognito contexts, but that also means no caching of resources, which also results in a major performance hit (based on our measures processing took twice the time).

https://crawlee.dev/api/browser-pool/interface/LaunchContextOptions#useIncognitoPages

We plan to overhaul this in the next major version and nea (which will likely be out in the first half of next year).

2 replies

slow-groovin Oct 29, 2024
Author

@B4nan Thank you, but I am still confused when not setting browserPerProxy: true,:

the session pool size is default 100, but the log shows the all requests have same session id: session_YKuZcs16NK
the log in requestHandler shows diff ports of proxyInfo , but in the access log of proxy, all these requests actually use the same one port : 50801, the first

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';



const startUrls = [
	'https://www.example.com',
	'https://example.net',
	'https://paste.c-net.org',
	'https://info.cern.ch',
	'https://www.example.org',
	'https://www.iana.org/domains/reserved',


	'https://www.example.com?q=1',
	'https://example.net?q=1',
	'https://paste.c-net.org?q=1',
	'https://info.cern.ch?q=1',
	'https://www.example.org?q=1',
	'https://www.iana.org/domains/reserved?q=1',

	'https://www.example.com?q=2',
	'https://example.net?q=2',
	'https://paste.c-net.org?q=2',
	'https://info.cern.ch?q=2',
	'https://www.example.org?q=2',
	'https://www.iana.org/domains/reserved?q=2',

];


function createProxyConfigurationFromJson() {
	return new ProxyConfiguration({
		newUrlFunction: (sessionId, options) => {
			const s = sessionId + ''
			const port = s.charCodeAt(s.length - 1) % 10 + 50800
			// console.log('newUrlFunction', sessionId, port, options?.request?.url)
			return `http://localhost:${port}`
		}
	})
}

const crawler = new PlaywrightCrawler({
	headless: false,
	maxRequestRetries: -1,
	navigationTimeoutSecs: 100,
	proxyConfiguration: createProxyConfigurationFromJson(),
	minConcurrency: 2,
	maxConcurrency: 5,
	maxRequestsPerMinute: 500,
	useSessionPool: true,
	persistCookiesPerSession: true,
	// launchContext: {
	// 	browserPerProxy: true,
	//   },
	preNavigationHooks: [
		async (context) => {
			const { page, log, session, proxyInfo, request } = context
			// console.log('preNavigationHooks', session?.id, session?.getCookieString(request.url), proxyInfo?.port, page.url(), request.url) //session id in here is same with requestHandler
		}
	],
	async requestHandler({ page, proxyInfo, session }) {
		console.log('requestHandler', session?.id, proxyInfo?.port, page.url())
		// await new Promise((resolve)=>{
		// 	setTimeout(()=>{
		// 		resolve(true)
		// 	},50_000)
		// })
	}
})
await crawler.run(startUrls);

log.txt

requestHandler session_YKuZcs16NK 50801 https://www.example.com/
requestHandler session_YKuZcs16NK 50809 https://example.net/
requestHandler session_YKuZcs16NK 50800 https://info.cern.ch/
requestHandler session_YKuZcs16NK 50803 https://www.example.org/
requestHandler session_YKuZcs16NK 50807 https://paste.c-net.org/
requestHandler session_YKuZcs16NK 50807 https://www.example.com/?q=1
requestHandler session_YKuZcs16NK 50806 https://example.net/?q=1
requestHandler session_YKuZcs16NK 50807 https://paste.c-net.org/?q=1
requestHandler session_YKuZcs16NK 50800 https://info.cern.ch/?q=1
requestHandler session_YKuZcs16NK 50804 https://www.example.org/?q=1
requestHandler session_YKuZcs16NK 50801 https://www.iana.org/domains/reserved
requestHandler session_YKuZcs16NK 50804 https://www.iana.org/domains/reserved?q=1
requestHandler session_YKuZcs16NK 50809 https://www.example.com/?q=2
requestHandler session_YKuZcs16NK 50807 https://example.net/?q=2
requestHandler session_YKuZcs16NK 50807 https://info.cern.ch/?q=2
requestHandler session_YKuZcs16NK 50807 https://www.example.org/?q=2
requestHandler session_YKuZcs16NK 50809 https://www.iana.org/domains/reserved?q=2
requestHandler session_YKuZcs16NK 50805 https://paste.c-net.org/?q=2

When settting browserPerProxy: true,, the behaviour is as expect, sessionID, browser instance, and proxyUrl form one-to-one correspondence.
Does it mean the guide chapters session-management proxy-management only be effective under setting browserPerProxy: true,?

B4nan Oct 29, 2024
Maintainer

I'd have to check the code, but I believe we keep the same session ID since you are using the same proxy, it would only make things more confusing if we keep rotating the session ID without actually changing the session/proxy behind the scenes.

only be effective under setting browserPerProxy: true,?

Well, that's a bit too general statement, those pages are not just about browsers and the proxy management guide is not just about crawlers. I will add some note in there, I guess we don't have this documented anywhere now. The plan was to get around this with the incognito mode, but once we implemented that, we saw the drastic perf hit, so we never swapped the default because of that.

slow-groovin · 2024-10-29T13:29:20Z

slow-groovin
Oct 29, 2024
Author

@B4nan Hi, in later testing of settting browserPerProxy: true,, I found another problem: one session ID corespond to multi proxy urls.

code

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';

const startUrls = []
for(let i=0;i<100;i++){
	startUrls.push(`https://example.net?q=${i}`)
}

const POOL_SIZE=10
function createProxyConfig() {
	return new ProxyConfiguration({
		newUrlFunction: (sessionId, options) => {
			const s = sessionId + ''
			const port = s.charCodeAt(s.length - 1) % POOL_SIZE + 50600
			return `http://localhost:${port}`
		}
	})
}

const crawler = new PlaywrightCrawler({
	headless: true,
	maxRequestRetries: -1,
	navigationTimeoutSecs: 100,
	proxyConfiguration: createProxyConfig(),
	minConcurrency: 5,
	maxConcurrency: 5,
	maxRequestsPerMinute: 500,
	useSessionPool: true,
	persistCookiesPerSession: true,
	sessionPoolOptions:{
		maxPoolSize:POOL_SIZE
	},
	launchContext: {
		browserPerProxy: true,
	},
	async requestHandler({ page, proxyInfo, session }) {
		console.log('requestHandler', session?.id, proxyInfo?.port, page.url())
	}
})
await crawler.run(startUrls);

log and parse code to find different proxy of same session

const input = `

requestHandler session_ep9wDhwkjb 50600 https://example.net/?q=0
requestHandler session_xRiNOaUl7v 50601 https://example.net/?q=1
requestHandler session_fJAOdWjM2m 50602 https://example.net/?q=2
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=3
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=4
requestHandler session_xRiNOaUl7v 50601 https://example.net/?q=8
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=9
requestHandler session_Q9IHaozP68 50605 https://example.net/?q=5
requestHandler session_Q9IHaozP68 50605 https://example.net/?q=6
requestHandler session_fJAOdWjM2m 50602 https://example.net/?q=11
requestHandler session_Q9IHaozP68 50605 https://example.net/?q=12
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=13
requestHandler session_0LJRAXgyAF 50606 https://example.net/?q=7
requestHandler session_xRiNOaUl7v 50601 https://example.net/?q=14
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=17
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=16
requestHandler session_Q9IHaozP68 50607 https://example.net/?q=10
requestHandler session_Q9IHaozP68 50607 https://example.net/?q=18
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=19
requestHandler session_Q9IHaozP68 50605 https://example.net/?q=20
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=22
requestHandler session_fJAOdWjM2m 50602 https://example.net/?q=23
requestHandler session_3s2UIf3g15 50608 https://example.net/?q=21
requestHandler session_3s2UIf3g15 50608 https://example.net/?q=15
requestHandler session_xRiNOaUl7v 50601 https://example.net/?q=24
requestHandler session_Q9IHaozP68 50605 https://example.net/?q=25
requestHandler session_ep9wDhwkjb 50600 https://example.net/?q=26
requestHandler session_fJAOdWjM2m 50602 https://example.net/?q=27
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=28
requestHandler session_xRiNOaUl7v 50601 https://example.net/?q=29
requestHandler session_fJAOdWjM2m 50602 https://example.net/?q=30
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=32
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=33
requestHandler session_3s2UIf3g15 50608 https://example.net/?q=35
requestHandler session_ep9wDhwkjb 50600 https://example.net/?q=34
requestHandler session_Q9IHaozP68 50605 https://example.net/?q=37
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=38
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=39
requestHandler session_xRiNOaUl7v 50601 https://example.net/?q=40
requestHandler session_gbneBPsejg 50609 https://example.net/?q=31
requestHandler session_gbneBPsejg 50609 https://example.net/?q=36
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=41
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=42
requestHandler session_3s2UIf3g15 50608 https://example.net/?q=43
requestHandler session_Q9IHaozP68 50607 https://example.net/?q=44
requestHandler session_0LJRAXgyAF 50606 https://example.net/?q=45
requestHandler session_0LJRAXgyAF 50606 https://example.net/?q=46
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=47
requestHandler session_0LJRAXgyAF 50606 https://example.net/?q=48
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=49
requestHandler session_Q9IHaozP68 50607 https://example.net/?q=50
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=51
requestHandler session_Q9IHaozP68 50607 https://example.net/?q=52
requestHandler session_Q9IHaozP68 50607 https://example.net/?q=53
requestHandler session_Q9IHaozP68 50605 https://example.net/?q=54
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=55
requestHandler session_Q9IHaozP68 50607 https://example.net/?q=56
requestHandler session_3s2UIf3g15 50608 https://example.net/?q=57
requestHandler session_xRiNOaUl7v 50601 https://example.net/?q=58
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=59
requestHandler session_Q9IHaozP68 50607 https://example.net/?q=60
requestHandler session_fJAOdWjM2m 50602 https://example.net/?q=61
requestHandler session_fJAOdWjM2m 50602 https://example.net/?q=62
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=63
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=64
requestHandler session_0LJRAXgyAF 50606 https://example.net/?q=65
requestHandler session_ep9wDhwkjb 50600 https://example.net/?q=66
requestHandler session_Q9IHaozP68 50607 https://example.net/?q=68
requestHandler session_Q9IHaozP68 50605 https://example.net/?q=67
requestHandler session_Q9IHaozP68 50605 https://example.net/?q=69
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=70
requestHandler session_ep9wDhwkjb 50600 https://example.net/?q=71
requestHandler session_fJAOdWjM2m 50602 https://example.net/?q=72
requestHandler session_0LJRAXgyAF 50606 https://example.net/?q=73
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=74
requestHandler session_ep9wDhwkjb 50600 https://example.net/?q=75
requestHandler session_xRiNOaUl7v 50601 https://example.net/?q=76
requestHandler session_xRiNOaUl7v 50601 https://example.net/?q=77
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=78
requestHandler session_Q9IHaozP68 50607 https://example.net/?q=79
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=81
requestHandler session_Q9IHaozP68 50605 https://example.net/?q=80
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=82
requestHandler session_0LJRAXgyAF 50604 https://example.net/?q=83
requestHandler session_ep9wDhwkjb 50600 https://example.net/?q=84
requestHandler session_fJAOdWjM2m 50602 https://example.net/?q=85
requestHandler session_0LJRAXgyAF 50606 https://example.net/?q=86
requestHandler session_ep9wDhwkjb 50600 https://example.net/?q=87
requestHandler session_ep9wDhwkjb 50600 https://example.net/?q=88
requestHandler session_3s2UIf3g15 50608 https://example.net/?q=89
requestHandler session_3s2UIf3g15 50608 https://example.net/?q=90
requestHandler session_gbneBPsejg 50609 https://example.net/?q=92
requestHandler session_xRiNOaUl7v 50601 https://example.net/?q=91
requestHandler session_3s2UIf3g15 50608 https://example.net/?q=93
requestHandler session_3s2UIf3g15 50608 https://example.net/?q=94
requestHandler session_ep9wDhwkjb 50600 https://example.net/?q=95
requestHandler session_gbneBPsejg 50609 https://example.net/?q=96
requestHandler session_gbneBPsejg 50609 https://example.net/?q=97
requestHandler session_fJAOdWjM2m 50602 https://example.net/?q=98
requestHandler session_M0mYt9wBNR 50603 https://example.net/?q=99
  `;

function parseLog(input: string) {
  const session2port: { [key: string]: Record<string,boolean> } = {};
  const port2session: { [key: string]: Record<string,boolean> } = {};

  const lines = input.trim().split('\n');
  lines.forEach(line => {
    const [_,sessionId,port] = line.trim().split(' ');

    if (!session2port[sessionId]) {
      session2port[sessionId] = {};
    }
    session2port[sessionId][port]=true;
    if(!port2session[port]){
      port2session[port]={}
    }
    port2session[port][sessionId]=true
  });

  return {port2session,session2port}
}


const result = parseLog(input);
console.log(result);

{
  port2session: {
    '50600': { session_ep9wDhwkjb: true },
    '50601': { session_xRiNOaUl7v: true },
    '50602': { session_fJAOdWjM2m: true },
    '50603': { session_M0mYt9wBNR: true },
    '50604': { session_0LJRAXgyAF: true },
    '50605': { session_Q9IHaozP68: true },
    '50606': { session_0LJRAXgyAF: true },
    '50607': { session_Q9IHaozP68: true },
    '50608': { session_3s2UIf3g15: true },
    '50609': { session_gbneBPsejg: true }
  },
  session2port: {
    session_ep9wDhwkjb: { '50600': true },
    session_xRiNOaUl7v: { '50601': true },
    session_fJAOdWjM2m: { '50602': true },
    session_M0mYt9wBNR: { '50603': true },
    session_0LJRAXgyAF: { '50604': true, '50606': true },
    session_Q9IHaozP68: { '50605': true, '50607': true },
    session_3s2UIf3g15: { '50608': true },
    session_gbneBPsejg: { '50609': true }
  }
}

1 reply

B4nan Oct 29, 2024
Maintainer

@barjin could you look into this one please?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

session id is diffrent in `newUrlFunction` and `requestHandler` #2726

{{title}}

Replies: 2 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

session id is diffrent in newUrlFunction and requestHandler #2726

slow-groovin Oct 28, 2024

Which package is this bug report for? If unsure which one to select, leave blank

Issue description

Code sample

Package version

Node.js version

Operating system

Apify platform

I have tested this on the next release

Other context

Replies: 2 comments · 3 replies

B4nan Oct 28, 2024 Maintainer

slow-groovin Oct 29, 2024 Author

B4nan Oct 29, 2024 Maintainer

slow-groovin Oct 29, 2024 Author

code

log and parse code to find different proxy of same session

B4nan Oct 29, 2024 Maintainer

session id is diffrent in `newUrlFunction` and `requestHandler` #2726

slow-groovin
Oct 28, 2024

I have tested this on the `next` release

Replies: 2 comments 3 replies

B4nan
Oct 28, 2024
Maintainer

slow-groovin Oct 29, 2024
Author

B4nan Oct 29, 2024
Maintainer

slow-groovin
Oct 29, 2024
Author

B4nan Oct 29, 2024
Maintainer