Skip to content

v1.2.0

Release v1.2.0: High-Speed Rate Limiting Optimizations
@A6h9lash A6h9lash tagged this 28 Oct 06:05 
 MAJOR PERFORMANCE IMPROVEMENTS:

New Rate Limiting Modes:
- FAST mode: 1000 jobs in ~11 minutes (ultra-fast)
- AGGRESSIVE mode: 1000 jobs in ~5 minutes (2x faster!)
- NORMAL mode: Standard rate limiting
- CONSERVATIVE mode: Safe delays for sensitive requests

Key Optimizations:
- Reduced base delays from 5s to 1.5s (3x faster)
- Smart retry logic with exponential backoff
- Optimized page delays: 0.5-1.0s (fast), 1.0-2.0s (aggressive)
- Enhanced 429 error handling with intelligent backoff
- Cap retry delays at 10-30s to prevent excessive waiting

Performance Results:
- AGGRESSIVE mode: 1000 jobs in ~5 minutes
- FAST mode: 1000 jobs in ~11 minutes
- Minimal 429 errors due to smart rate limiting
- Reliable large-scale scraping capabilities

New Features:
- rate_limit_mode parameter in scrape_jobs()
- 4 different rate limiting strategies
- Comprehensive retry mechanism
- Better error handling and logging

Usage:
from jobspy_enhanced import scrape_jobs

jobs = scrape_jobs(
    site_name=['linkedin'],
    search_term='data scientist',
    results_wanted=1000,
    rate_limit_mode='aggressive'  # Fastest mode
)

jobs = scrape_jobs(
    site_name=['linkedin'],
    search_term='data scientist',
    results_wanted=1000,
    rate_limit_mode='fast'  # Ultra-fast mode
)

Breaking Changes: None - fully backward compatible
Dependencies: No new dependencies required
Assets 2
Loading