How to run crawlers in a distributed manner

[fuergaosi233]

I've got a crawler running locally, but it only runs standalone, how do I get it to run distributed on my K8s cluster?

[fuergaosi233]

Does apify have a similar implementation or plan for wrapping a message queue into a RequestQueue?

[mnmkng]

The basics of running in Docker are explained in the Docker guide.

If you would like to use distributed storages or queue, you can implement an adapter / plugin and then inject it using the storageClient option of Crawlee's configuration.

There are three implementations of the storage client that you can use for reference. Apify Client, Apify Storage Local emulation and Crawlee Memory Storage.

Finally, you can use the Apify platform for storage and queue simply by getting a free account (to get a token) and then using the Apify Client as the storage client.

[fuergaosi233]

Are there other open source implementations of StorageClient available? I would like to refer to a new Client implementation

[mnmkng]

Not yet, unfortunately. We only added this feature - the pluggability - in v3, which is very recent. In v2 the storage clients were tightly coupled with this library. I guess it will take a few months before other StorageClient implementations will appear.