[desuarchive] 429 error appears repeatedly while scraping images of a thread, despite the original images being downloadable manually. #5129
-
Hello, I was scraping this thread on desuarchive, when I noticed the error "429 Too Many Requests" repeatedly showing up for a bunch of images I was scraping. It then stops but then started again occasionally, as seen with the image below, and the output txt file at the bottom. I thought the images themselves were undownloadable on the thread somehow, but when I check a random post and tried pressing on its image, I found that I was able to download its image, as seen below. The highlighted text is the name of the image. I have no special config relating to desuarchive, so Im not sure what could have caused these errors. I hope they can be resolved soon, thank you. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
You should try an artificial delay, by using the Try it with a low duration first, and gradually ramp it up until you see less of HTTP 429 errors, or none at all. |
Beta Was this translation helpful? Give feedback.
You should try an artificial delay, by using the
"sleep-request"
option, for example.Try it with a low duration first, and gradually ramp it up until you see less of HTTP 429 errors, or none at all.