feat(crawler): optional close page after screenshot #723
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Introduce a new
close_after_screenshotparameter in CrawlerRunConfig to provide more control over browser session management during screenshot capture. Update screenshot methods in AsyncPlaywrightCrawlerStrategy to conditionally close the browser page based on this new configuration option.List of files changed and why
crawl4ai/async_configs.py - To add the new close_after_screenshot parameter:
Added parameter to CrawlerRunConfig class
Added to init method
Added to from_kwargs method
Added to to_dict method
crawl4ai/async_crawler_strategy.py - To implement page closing logic:
Modified take_screenshot_scroller method to use close_after_screenshot parameter
Modified take_screenshot_naive method to use close_after_screenshot parameter
Modified _crawl_web method to pass close_after_screenshot to screenshot methods
docs/md_v2/core/browser-crawler-config.md - To document the new parameter:
Added documentation for close_after_screenshot parameter
Updated usage examples
examples/screenshot_example.py - To demonstrate the new parameter usage:
Added examples with and without automatic page closing
tests/test_screenshot_close.py - To create testcases
How Has This Been Tested?
Testing has been implemented through a dedicated test file test_to_remove.py which verifies several aspects of screenshot behavior:
Screenshot without page closure (test_screenshot_without_close)
Verifies that the page remains accessible after taking a screenshot when close_after_screenshot=False
Confirms successful screenshot capture
Validates that the page can be reused for subsequent operations
Screenshot with page closure (test_screenshot_with_close)
Tests behavior with close_after_screenshot=True
Confirms proper screenshot generation
Multiple screenshots in same session (test_multiple_screenshots_same_session)
⚠️ However, a new browser tab is unexpectedly opened even though the same session ID is used
Tests taking multiple screenshots within the same session
Verifies different close settings in sequence
Ensures proper resource management
The tests are run in sequence with proper initialization and cleanup:
Checklist:
Thanks for your usefull work and your approach of it !