[repo-monitor] Medium: Checkpoint on pause loses in-flight request results

[Liohtml]

Summary

When request_pause() is called, the crawl loop breaks immediately without waiting for in-flight tokio::spawn tasks to finish. The checkpoint is saved before those tasks complete, permanently losing any URLs they would have enqueued.

Location

• File: src/spiders/engine.rs
• Line(s): 147–219

Severity

Medium

Details

Active spawned tasks continue executing after the loop breaks. The checkpoint is saved immediately, but URLs enqueued by in-flight tasks are lost. A resumed crawl from the checkpoint will miss those URLs.

Suggested Fix

Wait for all active tasks to drain before saving the checkpoint:

// After loop break:
while self.active_tasks.load(Ordering::SeqCst) > 0 {
    tokio::time::sleep(Duration::from_millis(50)).await;
}
// Then save checkpoint

---

Automated finding by repo-monitor

[Liohtml]

Analysis

Confirmed. This is the most subtle data-loss bug in the repo.

src/spiders/engine.rs — the crawl loop breaks on paused == true immediately, then CheckpointManager::save() is called. But by the time save() runs, in-flight tokio::spawn tasks are still running and may enqueue new URLs into the scheduler. Those enqueued URLs are lost — they're in the scheduler after save() completes only if the task happened to finish before that point, which is a race condition.

What's actually lost on pause:

• URLs that in-flight tasks would have discovered (via Spider::parse() return values)
• Items that were about to be collected
• The URL fingerprints for in-flight requests (so they'll be re-fetched on resume)

The fix: Drain active_tasks to zero before saving the checkpoint.

// Before saving checkpoint:
while active_tasks.load(Ordering::SeqCst) > 0 {
    tokio::time::sleep(Duration::from_millis(50)).await;
}
// Now save — all spawned tasks have completed and their enqueues are in the scheduler
checkpoint_manager.save(&scheduler).await?;

Also fix the seen-set issue: The checkpoint currently saves only the pending queue, not the seen fingerprints set. On resume, all restored requests have dont_filter = true (bypass dedup). This means if a crawl was paused and resumed, any URL that was in the queue at pause time will be re-fetched even if it was already scraped in a previous session. The fix is to serialize Scheduler's HashSet<fingerprint> into the checkpoint JSON alongside the pending queue.

Effort: 1-2 hours to fix both the drain and the seen-set serialization correctly.