Skip to content

v0.10.0

Choose a tag to compare

@dchuk dchuk released this 25 Feb 06:37
· 283 commits to main since this release
cf278be

Added

  • Maintenance queue for non-fetch jobs. New third queue (source_monitor_maintenance) separates non-time-sensitive jobs from the fetch pipeline. Health checks, cleanup, favicon fetching, image downloading, and OPML import jobs now run on the maintenance queue. Configure via config.maintenance_queue_name and config.maintenance_queue_concurrency.
  • Configurable scheduler batch size. config.fetching.scheduler_batch_size (default 25, was hardcoded at 100) controls how many sources are picked up per scheduler run.
  • Configurable stale fetch timeout. config.fetching.stale_timeout_minutes (default 5, was hardcoded at 10) controls how long a source can remain in "fetching" status before reset.
  • Stagger fetch times rake task. source_monitor:maintenance:stagger_fetch_times distributes overdue sources across a configurable time window.

Fixed

  • Fetch pipeline error handling safety net. DB update failures propagate; broadcast failures rescued; ensure block guarantees status reset. FollowUpHandler rescues per-item scrape enqueue failures.
  • Fixed-interval sources now get scheduling jitter. ±10% jitter on next_fetch_at, preventing thundering herd effects.
  • ScheduleFetchesJob uses configured batch size instead of legacy constant.

Changed

  • Default scheduler batch size reduced from 100 to 25.
  • Default stale fetch timeout reduced from 10 to 5 minutes.
  • 7 jobs moved from fetch queue to maintenance queue: SourceHealthCheckJob, ImportSessionHealthCheckJob, ImportOpmlJob, LogCleanupJob, ItemCleanupJob, FaviconFetchJob, DownloadContentImagesJob.

Upgrade Notes

  • Action required: Update your solid_queue.yml to include the new maintenance queue.
  • Run bin/rails source_monitor:maintenance:stagger_fetch_times after upgrading if many sources are overdue.
  • See upgrade guide for full details.