max context

[maxwill-max]

is there any plan to implement max context for concurrency and proxy purposes? it will help a lot when dealing with multiple isolated sessions of same domain

[elacuesta]

Do you mean a way to set an upper limit on the amount of contexts, like the PLAYWRIGHT_MAX_PAGES_PER_CONTEXT setting limits the amount of pages within a context? What's your use case, can you elaborate?

[maxwill-max]

hi i just learning playwriight and moving from splash. playwright is super great so far. i am trying to scrape a website with multiple sessions and one proxy each session, its kinda like using cookiejar in splash. while pages in context is great, pages share resources including cookies, proxies, useragents with other pages in same context so it wont help creating more than session in a same context. I meant more like PLAYWRIGHT_MAX_CONTEXT like CONCURRENT_REQUESTS in scrapy

[maxwill-max]

hi, what i mean is that, to set maximum number of concurrent context(not pages)

[elacuesta]

I understand now, thanks for the clarification. Unfortunately, this is not a priority at the moment. Keeping the code simple is one of my main goals here, and I'm not sure this is a problem frequent enough to justify a built-in solution.
There are existing ways to dynamically create/destroy contexts (https://github.com/scrapy-plugins/scrapy-playwright#multiple-browser-contexts), that should be enough for most cases I think. You could also implement a solution based on semaphores, similar to the PLAYWRIGHT_MAX_PAGES_PER_CONTEXT setting approach.

[elacuesta]

Seems like this won't be as cumbersome as I thought it could be. I will have an update shortly.

[maxwill-max]

very glad to hear it. thank you