purgeDefaultStorages() doesn't work with express

[soupman99]

I'm running a crawler that is called via an expressjs call.

When I call the same route again, my crawler runs again but shows that all routes have already finished. I'm even removing the './storage' folder

I read the documentation but can't seem to get the purgeDefaultStorages() to work.

How would I go about "resetting" crawlee so that there's not cached results?

[B4nan]

How does your express endpoint look like?

[metalwarrior665]

I'm trying to debug it now a bit. It clears the file storage correctly but something is lingering in memory that keeps it cached.

[gumarm]

I don't know if it helps you out, but even using the MemoryStorage class as the storageClient has this issue. I had to disable all files on disk for it to work well on gcloud functions (saw a similar discussion for AWS Lambdas: #1612).

[metalwarrior665]

The memory storage is still persisting to disk by default in an interval so there isn't really any change with using fully disk-synced storage. The problem seems that even if the filesystem is clean, something in memory is still caching the previous requests used in a different crawler.

[metalwarrior665]

Btw this code works, it is just a bit more verbose. I would still consider the original complaint as a bug though. Something is persisted on the global RequestQueue instance or it gives you the same instance if you spawn new crawler or whatever.

const getNewCrawler = (requestQueue: RequestQueue) => new CheerioCrawler({
    requestQueue,
    requestHandler: async ({ request, $ }) => {
        console.log(`Processing ${request.url}...`);
        await Dataset.pushData({ title: $('title').text() });
    },
});

const requestQueue1 = await RequestQueue.open();
await requestQueue1.addRequest({ url: 'https://www.google.com' });

const crawler = getNewCrawler(requestQueue1);

await crawler.run();

await requestQueue1.drop();

// await Configuration.getStorageClient().purge();

const requestQueue2 = await RequestQueue.open();
await requestQueue2.addRequest({ url: 'https://www.google.com' });

const crawler2 = getNewCrawler(requestQueue2);

await crawler2.run();

await requestQueue2.drop();

[gumarm]

@metalwarrior665 You're awesome, this workaround works !

Thanks a lot for looking into this, it's much appreciated :)

[soupman99]

@gumarm someone posted this to my question on stackoverflow but haven't tested it yet. If you test and get it working can you reply back so I know it works?

https://stackoverflow.com/a/74925954?noredirect=1

[gumarm]

I was already using CRAWLEE_PERSIST_STORAGE: false, so I added the CRAWLEE_PURGE_ON_START: false process env and I'm still getting the same broken behaviour.

Just to be sure, I duplicated the same configs directly in my crawler config, but still nothing.

That answer doesn't seem to do much unfortunately...

[soupman99]

K. Thanks for testing it. I guess we'll just have to wait for an update. I tried everything I can think of too and all teh suggestions, but still no dice.

The one thing I did get to work was to split it into it's own JS file and fork that process in my code. It's not ideal (makes me a little nervous to do that) but it could be made to be more robust.

[gumarm]

Oh, yeah, given I'm running this in a gcloud function I'm not too keen on doing it with a fork since I don't control the runtime fully.

@B4nan Got anymore ideas ?

Btw, I'm using @crawlee/http with something like this, which runs every 5 minutes for the same URLs:

export const getData = () => {
  cont results = []
  const crawler = new HttpCrawler(
    {
      additionalMimeTypes: ["application/json"],
      maxConcurrency: 2,
      maxRequestsPerMinute: 30,
      maxRequestRetries: 1,
      async requestHandler({ request, response, body }) {
        // Extract some data from response
        // Push to "results" array in main function body scope
      },
    },
    // Trying everything I can to reset state between runs
    new Configuration({
      persistStorage: false,
      purgeOnStart: true,
      storageClient: new MemoryStorage({
        persistStorage: false,
      }),
    })
  );

  await crawler.addRequests(getUrls()); // generates 1-4 requests
  await crawler.run();

  // we need to purge storage for the next run otherwise the crawler caches requests (this doesn't work reliably)
  const storageClient = crawler.config.getStorageClient();
  if (storageClient.purge && storageClient.teardown) {
    await storageClient.purge();
    await storageClient.teardown();
  }

  return results
}

[soupman99]

@gumarm yeah I'm using gcloud too. You raise an even more compelling argument against what I suggested. If I find a solution I'll try to remember to update this ticket.

[aikin-vip]

Same problem, however I used another solution to temporarily bypass this problem, for example:

await crawler.addRequests([{url:'http://127.0.0.1',uniqueKey:'request-id'+Date.now()}])

This is not the best solution ideally, but can be temporarily bypassed pending an official fix