Discussion: Store state for fault-tolerance crawler

[ghost]

I think Postgres would be a good way to store the spider state to in case the system crash it can continue the crawling from there it was stoped.

Is there any recommendations or suggestions how to implement it in my current Crawly project?

[Ziinc]

You can use a custom pipeline to update your database tables with your scraped data.

When your spiders are started, your starting urls can then be queried from the database.

You can use an ecto-based job queue like Honeydew to poll your database for things to be scraped and startup the relevant spiders accordingly. Or you can use Mnesia to persist state.

[ghost]

@Ziinc Thanks again. Soon will do.

[lucaong]

Pitching in to mention CubQ as a way to implement durable queues in an embedded database. Full disclaimer: I am the author of the library. I created it mostly for embedded software scenarios, but I think it would fit well for keeping a crawler queue too.

[Ziinc]

For improved fault tolerance, usage of persistent queues for requests/ScrapedItems would definitely be good.

However, this would involve an additional dependency, and it would be hard to argue for CubQ (backed by CubDb) over other more established queue libraries backed by Mnesia.

[Ziinc]

I don't think that there is a strong case for improving fault tolerance and stability now when there are other features to be implemented.

Perhaps in the future, we can reopen this when there is a v1.