Skip to content

Conversation

@alexandreolives
Copy link

@alexandreolives alexandreolives commented Feb 20, 2025

Summary

Introduce a new close_after_screenshot parameter in CrawlerRunConfig to provide more control over browser session management during screenshot capture. Update screenshot methods in AsyncPlaywrightCrawlerStrategy to conditionally close the browser page based on this new configuration option.

List of files changed and why

  1. crawl4ai/async_configs.py - To add the new close_after_screenshot parameter:
    Added parameter to CrawlerRunConfig class
    Added to init method
    Added to from_kwargs method
    Added to to_dict method

  2. crawl4ai/async_crawler_strategy.py - To implement page closing logic:
    Modified take_screenshot_scroller method to use close_after_screenshot parameter
    Modified take_screenshot_naive method to use close_after_screenshot parameter
    Modified _crawl_web method to pass close_after_screenshot to screenshot methods

  3. docs/md_v2/core/browser-crawler-config.md - To document the new parameter:
    Added documentation for close_after_screenshot parameter
    Updated usage examples

  4. examples/screenshot_example.py - To demonstrate the new parameter usage:
    Added examples with and without automatic page closing

  5. tests/test_screenshot_close.py - To create testcases

How Has This Been Tested?

Testing has been implemented through a dedicated test file test_to_remove.py which verifies several aspects of screenshot behavior:

  1. Screenshot without page closure (test_screenshot_without_close)
    Verifies that the page remains accessible after taking a screenshot when close_after_screenshot=False
    Confirms successful screenshot capture
    Validates that the page can be reused for subsequent operations

  2. Screenshot with page closure (test_screenshot_with_close)
    Tests behavior with close_after_screenshot=True
    Confirms proper screenshot generation

  3. Multiple screenshots in same session (test_multiple_screenshots_same_session)
    Tests taking multiple screenshots within the same session
    Verifies different close settings in sequence
    Ensures proper resource management
    ⚠️ However, a new browser tab is unexpectedly opened even though the same session ID is used

The tests are run in sequence with proper initialization and cleanup:

  • Browser is initialized once before all tests
  • Each test runs in isolation
  • Resources are properly cleaned up after all tests complete
  • Error handling is implemented for each test case

Checklist:

  • ✅ My code follows the style guidelines of this project
  • ✅ I have performed a self-review of my own code
  • ✅ I have commented my code, particularly in hard-to-understand areas
  • ✅ I have made corresponding changes to the documentation
  • ✅ I have added/updated unit tests that prove my fix is effective or that my feature works
  • ✅ New and existing unit tests pass locally with my changes

Thanks for your usefull work and your approach of it !

Introduce a new `close_after_screenshot` parameter in CrawlerRunConfig to provide more control over browser session management during screenshot capture. Update screenshot methods in AsyncPlaywrightCrawlerStrategy to conditionally close the browser page based on this new configuration option.
…ture

Enhance documentation and test coverage for the new `close_after_screenshot` parameter:
- Create a new example script demonstrating screenshot behavior
- Update browser-crawler-config documentation with usage details
- Add comprehensive test suite for screenshot closing functionality
@alexandreolives alexandreolives changed the title feat(crawler): optional close browser after screenshot feat(crawler): optional close page after screenshot Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant