docs/introduction/saving-data

[giscus[bot]]

docs/introduction/saving-data

Your first steps into the world of scraping with Crawlee

https://crawlee.dev/docs/introduction/saving-data

[tteguayco]

In the case we want to save the scraped data into a database (.e.g Postgres, Mongo), does Crawlee incorporate anything for it or should we add our own custom code around the framework to handle such scenario?

[B4nan]

The storage layer is abstracted and you can implement your own, but the abstraction is a bit complex and there is no documentation around this currently. You could maybe try to extend the memory storage implementation, and override only the dataset methods. But it will be much simpler to do it outside of crawlee and just use your own helpers to store the data instead of the crawlee's dataset abstraction.

Going forward, we would like to simplify the storage interface some time in future (maybe v4), so the extension is simpler and maybe even provide more native storage implementations.

[tteguayco]

That sounds really nice, thanks!

[dbjpanda]

Hi, I am just wondering if it is now possible to implement a custom storage interface? Could not find any docs.