How to re-run - later - 403 responses?

[charnould]

Hello,

First, thanks a lot for this lib; I've learnt a lot.
(I've tried crawling before... but without a lot of success doing simple GET requests; even if it's working sometimes!)

I'm currently trying to scrape a website for some titles, but run into a few issues.

1. This website doesn't like a lot scrapping
   So to avoid bothering it, i run with very low maxConcurrency (5) and maxRequestsPerMinute (10), and only some url per day.

2. However, at the end of my crawl, I've got some Reclaiming failed request back to the list or queue. Request blocked - received 403 status code. Even if I've tried to read carefully Crawlee docs, I didn't find how to re-run (later of course) these failing URLs. I hope/think that they are somewhere in my storage folder...

Is there a command to re-run only the failing links/request?
Thanks a lot for your help.

[metalwarrior665]

There isn't unfortunately yet a simple command. It is implemented for the Apify platform queue - https://apify.com/lukaskrivka/rebirth-failed-requests. So you could reuse that code locally, it iterates over all requests and resets those that have more errorMessages than retries (because last fail doesn't retry).

But I'm not sure now (I'm not the core maintainer) if the same client methods are available for filesystem queue.

[charnould]

I've made a simple script getting 403 and relative URLs for rerun.
Thanks.