Make each collection step it's own checkpoint. That is, a collection manifest shard should be saved after each collection step. The status of each step should be saved in the manifest too (just a bool flag with "se_ran", "website_ran", etc). Then, when the collection step is re-run, the shard is read in and compared to user config. If a user-enabled step has completed, skip it and do the next step. If all steps are complete, no more processing required for that step.
Make each collection step it's own checkpoint. That is, a collection manifest shard should be saved after each collection step. The status of each step should be saved in the manifest too (just a bool flag with "se_ran", "website_ran", etc). Then, when the collection step is re-run, the shard is read in and compared to user config. If a user-enabled step has completed, skip it and do the next step. If all steps are complete, no more processing required for that step.