feat(session-pool): add sessionReuseStrategy option - #3587
Conversation
Introduces 'random' (default), 'round-robin', and 'use-until-failure' strategies for picking sessions from the pool. Non-random strategies reuse existing sessions before creating new ones, fixing the issue where sessions were never reused until the pool reached maxPoolSize.
There was a problem hiding this comment.
Pull request overview
Adds a new SessionPoolOptions.sessionReuseStrategy option to control how sessions are selected/reused from the pool, addressing issue #3270’s request for configurable reuse behavior.
Changes:
- Introduces
sessionReuseStrategyoption with'random' | 'round-robin' | 'use-until-failure'and default'random'. - Updates
SessionPoolselection logic to implement the new reuse strategies. - Adds tests and updates session management documentation to describe the strategies.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
packages/core/src/session_pool/session_pool.ts |
Adds the new option/type and implements strategy-based session picking. |
test/core/session_pool/session_pool.test.ts |
Adds coverage for the three reuse strategies’ basic behaviors. |
docs/guides/session_management.mdx |
Documents the new sessionReuseStrategy option and its intended use cases. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…nd-robin - replace intermediate usable[] array with inline iteration (O(1) space) - random and round-robin now return undefined when any session in a full pool is retired, letting getSession evict and replenish as before - use-until-failure skips retired sessions and returns the first usable one - fix round-robin JSDoc to reflect the actual fill-first behaviour
## Summary Simplified and optimized the session selection logic in `SessionPool.getSession()` by consolidating duplicate code paths and improving the random selection algorithm. ## Key Changes - **Unified 'use-until-failure' strategy**: Replaced explicit loop with `Array.find()` for cleaner code - **Consolidated round-robin and random strategies**: Both now follow the same pattern of selecting a session and then checking if it's usable, eliminating duplicate usability checks - **Improved random selection**: Replaced reservoir sampling algorithm with a simpler `_getRandomIndex()` method call, reducing algorithmic complexity - **Removed redundant cleanup logic**: Eliminated the separate loop that checked for retired sessions in full pools; cleanup now happens naturally when an unusable session is returned - **Enhanced test coverage**: Updated test to call `getSession()` 50 times instead of once to better validate session reuse behavior under repeated access patterns ## Implementation Details The refactoring maintains the same external behavior while reducing code duplication. The key insight is that for round-robin and random strategies, we can select a session first and then check its usability, rather than iterating through all sessions to find a usable one. This simplifies the logic and makes the code more maintainable.
| Please, bear in mind that a Session pool needs time to find working IPs and build up the pool, | ||
| so we will probably see a lot of errors until it becomes stabilized. | ||
|
|
||
| ## Session reuse strategy |
There was a problem hiding this comment.
One more strategy that I had in the first version of SessionPool (pre-Crawlee) is to not fill the pool first but have proportional chance to pick an existing session. E.g. you have maxPoolSize: 100, the first pick is 100% new session, the second pick is 99% new session vs 1 % old session etc.
The advantage is that you start reusing working sessions earlier and those are more likely to work than new ones. I think practically the difference vs current is very small so might not be a reason to add it, but just want to keep it in the back of your head if similar case comes in.
There was a problem hiding this comment.
Interesting, this basically looks like random, if the SessionPool was prefilled before use, right?
I think practically the difference vs current is very small
I cannot speak on that matter, but it does sound a little niche right now. I'll keep it in my mind, should this ever become relevant, though. Thank you for sharing 👍
Adds `SessionPoolOptions.sessionReuseStrategy` option for better control over `Session` retrieval. Closes #3270
Adds
SessionPoolOptions.sessionReuseStrategyoption for better control overSessionretrieval.Closes #3270