Skip to content

v1.5.0: Scheduled Scraping & Automation

Choose a tag to compare

View all tags
@doublegate doublegate released this 02 Oct 01:04
· 84 commits to main since this release
v1.5.0
cb48e68

WebScrape-TUI v1.5.0: Scheduled Scraping & Automation

📅 Major Feature Release

This release introduces a comprehensive scheduled scraping system with background automation, allowing users to configure recurring scrapes that run automatically using cron-like scheduling.

🎯 Key Features

Scheduled Scraping System

  • ✅ Create, manage, and delete scheduled scrapes
  • ✅ Background scheduler using APScheduler library
  • ✅ Automatic execution at specified intervals
  • ✅ Enable/disable schedules without deletion
  • ✅ Comprehensive status tracking (last run, next run, run count, status)
  • ✅ Error tracking and logging for failed scrapes

Schedule Types

  • Hourly: Execute every hour on the hour
  • Daily: Execute at specific time (HH:MM format, e.g., 09:00)
  • Weekly: Execute on specific day and time (day:HH:MM format, e.g., 0:09:00 for Monday 9am)
  • Interval: Execute every N minutes (custom interval in minutes)

Schedule Management Modal (Ctrl+Shift+A)

  • DataTable showing all schedules with comprehensive status
  • Add new schedules with intuitive form interface
  • Enable/disable schedules with single button click
  • Delete schedules with confirmation dialog
  • Real-time schedule status updates

🔧 Technical Implementation

  • ScheduleManager Class: CRUD operations, execution recording, next run calculations
  • Background Scheduler: APScheduler integration with automatic loading on startup
  • Schedule Execution: Automatic scrape execution with status tracking and error capture
  • Database Schema: New scheduled_scrapes table with foreign key constraints
  • Modal Screens: ScheduleManagementModal and AddScheduleModal with validation

📦 Dependencies

  • Added APScheduler>=3.10.0 for background task scheduling

✅ Testing

  • 16 new tests in test_scheduling.py
  • 126 total tests (all passing)
  • Comprehensive coverage for CRUD operations and schedule calculations

⌨️ Keyboard Shortcuts

  • Ctrl+Shift+A: Open Schedule Management modal
  • Ctrl+.: Open Settings modal (changed from Ctrl+,)

🚀 Use Cases

  • Daily News Aggregation: Schedule HN, Reddit, and tech news scraping every morning
  • Hourly Price Monitoring: Track product prices every hour for deals
  • Weekly Report Compilation: Gather industry articles every Monday morning
  • Custom Intervals: Scrape data every 15, 30, or 60 minutes
  • Automated Data Collection: Hands-free data gathering for research projects

📄 Full Changelog

See CHANGELOG.md for detailed feature list and technical details.

This release transforms WebScrape-TUI into a fully automated data collection platform! 🎉

Assets 2
Loading