Turn off storage

[tsrdatatech]

I am going to be running some crawlers as Lambdas on AWS and need to turn the storage off so it does not cause issues for me. I looked at the docs but did not see anything on how to turn that off. I am talking about the kay-value and requestque that is written. How do I turn that off?

[B4nan]

You cant disable the storage layer, it drives the crawler, but I guess you just want to disable the state persistence to JSON files and keep things only in memory, right?

I dont think we have an easy way to do that right now. You could definitely extend @crawlee/memory-storage and adjust the code to not write anything on disk, and pass this custom storage implementation via config.useStorageClient(new MyStorage()).

We should add an option for this @vladfrangu

[vladfrangu]

I can look into adding an option for this 👍

[tsrdatatech]

Thanks, that's what I was looking for was a way to not write to disk for this. I do think there should be an option for this in these types of cases.

[tsrdatatech]

I am trying to extend the memory-storage as you have suggested but am at a loss for how to actually implement the parts needed to avoid the writing of the JSON files. I seen there are clients at https://github.com/apify/crawlee/tree/master/packages/memory-storage/src/resource-clients that do the writing but I am not sure how I extend those to not write out the jsons. Can you give me an idea on how to do this part?

[B4nan]

We should have a new version that will support this out of box soon, probably as early as tomorrow. The PR is here if you are interested in what parts needed adjustments - the worker-utils.ts is the file that actually writes to disk.

[tsrdatatech]

awesome @B4nan thanks for the update.