Add pause, stop, resume

[sjdirect]

Add functionality that will allow a crawl to be continued from where it was stopped or paused.

[sjdirect]

A few ideas...

var pausedCrawler = manager.Pause(crawler);
manager.Resume(pausedCrawler);

[sjdirect]

Made all classes in Abot project Serializable so others may implement a pause/resume

[sjdirect]

More talk about this here...
https://groups.google.com/forum/#!topic/abot-web-crawler/KiYhgjaESNU

[mng-au]

Hi sjdirect, if you don't mind I ask, I remember you have a file based Url and Crawl repository in Google repository before, or is it my memory corrupted or fragmented. I hope I understand this issue correctly, but as long as Abot resume previous unfinished task, that will satisfy this ticket? i.e. stop / resume and pause is consider as stop. I am working on a simple Mongodb based Scheduler for Abot, I will upload to Github later, but then again, if you don't mind it is in F#. Thanks.

[sjdirect]

Yes there use to be a file based crawl repo but it was overly complex and under performant. A lightweight version of it MAY be created if I decide that is the best way to pause the crawl.

[mng-au]

Hi Steven, I added a new repository using Redis as scheduler store (I thought about using MongoDB before, but Redis gives much better performance as it runs in memory). That should allow the crawler to start and stop without losing track of previous progress. Hope this will help. Cheers. https://github.com/mnta/Abot.Redis.Scheduler

[sjdirect]

Thanks, i'll take a look at this when I cross that bridge

On Sat, Oct 17, 2015 at 5:18 AM, mnta notifications@github.com wrote:

Hi Steven, I added a new repository using Redis as scheduler store (I
thought about using MongoDB before, but Redis gives much better performance
as it runs in memory). That should allow the crawler to start and stop
without losing track of previous progress. Hope this will help. Cheers.
https://github.com/mnta/Abot.Redis.Scheduler

—
Reply to this email directly or view it on GitHub
#45 (comment).

[sjdirect]

You can use the Pause/Resume feature of AbotX. AbotX build on top of abot. Closing issue.