v2.41.1

v2.41.0 - WebDriver Support

This release adds WebDriver support via the thirtyfour crate, enabling browser
automation using the W3C WebDriver protocol. Connect to remote Selenium Grid,
chromedriver, geckodriver, and more.

New Features

WebDriver Integration

• Automatic crawl integration: When webdriver_config is set, crawl()
  automatically uses WebDriver instead of raw HTTP
• Multiple browser support: Chrome, Firefox, and Edge
• Remote server support: Connect to Selenium Grid, BrowserStack, or any W3C
  WebDriver-compatible server
• Stealth mode: Navigator property overrides to reduce automation detection

New Feature Flags

Flag	Description
webdriver	Core WebDriver support via thirtyfour
webdriver_headed	Run browser in headed (visible) mode
webdriver_stealth	Enable stealth mode to hide automation
webdriver_screenshot	Screenshot capture support
webdriver_chrome	Chrome-specific features
webdriver_firefox	Firefox-specific features
webdriver_edge	Edge-specific features

Usage

use spider::website::Website;                                                   
use spider::features::webdriver_common::{WebDriverConfig, WebDriverBrowser};    
                                                                                
let mut website = Website::new("https://example.com")                           
    .with_webdriver(                                                            
        WebDriverConfig::new()                                                  
            .with_server_url("http://localhost:4444")                           
            .with_browser(WebDriverBrowser::Chrome)                             
            .with_headless(true)                                                
    )                                                                           
    .build()                                                                    
    .unwrap();                                                                  
                                                                                
// crawl() automatically uses WebDriver                                         
website.crawl().await;                                                          
                                                                                
Examples                                                                        
                                                                                
Three new examples added:                                                       
- webdriver - Basic WebDriver crawling                                          
- webdriver_remote - Remote Selenium Grid connection                            
- webdriver_screenshot - Screenshot capture                                     
                                                                                
# Start chromedriver                                                            
chromedriver --port=4444                                                        
                                                                                
# Or use Selenium Grid with Docker                                              
docker run -d -p 4444:4444 --shm-size="2g" selenium/standalone-chrome:latest    
                                                                                
# Run example                                                                   
cargo run --example webdriver --features="webdriver webdriver_stealth"          
                                                                                
Full Changelog                                                                  
                                                                                
https://github.com/spider-rs/spider/compare/v2.40.5...v2.41.1