Expose config option to not persist SessionPool and Statistics

[metalwarrior665]

Which package is the feature request for? If unsure which one to select, leave blank

None

Feature

This is a recurring issue that people don't want extra files to be added or extra Apify API calls being triggered by this. So we should probably have flags to easily disable these, currently you have to hack the internal APIs.

Motivation

as above

Ideal solution or implementation, and any additional constraints

...

Alternative solutions or implementations

No response

Other context

No response

[B4nan]

So you'd like to disable this on apify platform, right? Because with memory storage you can already disable persistence completely and keep things only in memory, but this should be more finegrained?

Can you maybe provide some of the hackish solutions you can use right now?

[metalwarrior665]

Yeah, the reason would be to make it more fine-grained for cases where you e.g. want to persist queue and session but not stats (example).

I was using

import {Statistics} from 'crawlee';
Statistics.prototype.persistState = () => Promise.resolve();

The use-case was that I was running helper Crawler in the background and I didn't want it to store any extra files. Another option would be to allow disabling the stats altogether (no log, no persist).

The last trigger for this was some user complaining that he has too many KV writes after running tons of small actors because of the persisting :)

Nothing super important but wanted to thin how to kill 2 birds (local and platform concerns) with 1 stone