How to schedule the spider to run daily job on 1am and are there any duplicate content check?

[bruceliu2008]

Hi,
I need to run the spider everyday on 1am or some specific time, are there any schedule available for this?

Another question is that are there any content duplicate check? for example, I do crawling everyday for website www.abc.com/aa.html for its xpath '/html/body/div[3]/div/div[2]/section', but if the content of '/html/body/div[3]/div/div[2]/section' is exactly the same as my last crawling, then I will just ignore it.

Thank you.

[zlzforever]

1. In windows: windows task scheduler
2. In linux: cron
3. See codes in DotnetSpider.Sample.Program: Startup.Run("-s:MultiSupplementSpider", "-tid:1", "-i:guid");
4. If you have a lots of machines, may you can try DotnetSpider.Enterprise.
5. No content duplicate check. I think you can add a unique column to save the content's md5.

[zlzforever]

closed. Open it if you need support again.