How to avoid memory overflow?

[dawangraoming]

I have a request queue of 100,000 items, and both Dataset and KeyValueStore are enabled

Dataset is used to store request results
KeyValueStore is used to record crawled URLs

 const dataset = await Dataset.open(`result_${jobId}`);
 const store = await KeyValueStore. open(`url_${jobId}`);
 const requestList = await RequestList.open(
    null,
    getRequestListWithLimitStringList(keywords),
  );

 const crawler = new HttpCrawler({
    maxRequestRetries: 3,
    maxConcurrency: 10,
    requestHandlerTimeoutSecs: 60,
    maxRequestsPerMinute: 50,
    requestList,
    async requestHandler({ request, body, crawler }) {
      const { keyword } = request.userData;
      const url = request.url;
      const result = JSON.parse(body.toString());

      const storeData = {
        url,
        keyword,
        result,
        date: dayjs().format('YYYY-MM-DD HH:mm:ss'),
      };
      await dataset.pushData(storeData);
      await store.setValue(keyword, {
        keyword,
        total: result.total_count,
      });
    },
  });

When crawling to 10,000 items, warnings of memory overload frequently appear, and the concurrency is automatically limited to 1

Is there any way to solve it?

---

Env version

crawlee : 3.1.0
node : v16.16.0

[LeMoussel]

What is getRequestListWithLimitStringList?
See in RequestList.open: RequestList represents a list of URLs to crawl, which is always stored in memory.

[dawangraoming]

getRequestListWithLimitStringList function just converts ["a", "b", ..., "z"] to ["https://url.com/a", ... , "https://url.com/a"]

But if it is because it is stored in memory, will the memory usage of nearly 1GB be caused by just 100,000 URLs using RequestList?

[dawangraoming]

I guess it should be the reason of Dataset, when I try to clear the local Dataset and restart, the memory is only a few hundred M.

As Dataset creates more and more disk files, memory also increases.

Looking at the results of the memory snapshot, consistent with my guess, Dataset takes up 86.94% of the memory.

[dawangraoming]

Is this because every time you pushData to the Dataset, the Data will be kept in memory?

If I only want to store locally and not keep in memory, is it possible?

[B4nan]

cc @vladfrangu, how does this work with memory storage? I guess we should respect the persistence, and as long as the data gets persisted to JSON, we don't need to hold it in the memory (but we can't do that when persistence is disabled)

[vladfrangu]

Uhh, currently it keeps it in memory and also flushes to disk, I suppose we can look into implementing a setting for this or making persist not store in RAM

[LeMoussel]

@dawangraoming While waiting for a solution, you can use Level - an Key–value database on disk - instead Dataset.

[dawangraoming]

Thank u 🥳