For the past day or so I've been having issues making it far into a run in Rule34.xxx due to Cloudflare challenges. I've been mostly doing what I call maintenance runs, which is when I already downloaded a tag a while back but now I'm running it again to grab the new stuff as well as older posts that have had the tag added to them.
It really depends how far I make it, but something I noticed is that it aligns more or less to when I would have received a 429 notice. With my usual settings I would get 429s at a certain frequency, so it's easy to see in the output.
I followed the usual steps to get around Cloudflare (#4844) but it's still happening. Doesn't help that https://api.rule34.xxx/ doesn't list rate limits so shooting in the dark, I'm currently increasing the sleep times in a trial and error fashion, but I wonder if there's anything else that could be done, it's currently my biggest bottleneck among the 14 boorus I download regularly (my script downloads the same tag from all boorus).
Something I notice when the Cloudflare error appears is that when opening the post URL that triggered it in the browser, most times I don't see a CF challenge to solve, and the post loads as expected. A few times a CF challenge appeared but it autosolved itself and redirected me to the post immediately.
Current relevant settings:
{
"rule34": {
"skip": true,
"tags": true,
"notes": true,
"sleep": 2.0,
"sleep-request": 15.0,
"sleep-429": 500,
"cookies": "~/.config/gallery-dl/cookies/rule34.txt",
"cookies-update": true,
"user-id": "xxxxxx",
"api-key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
}
For the past day or so I've been having issues making it far into a run in Rule34.xxx due to Cloudflare challenges. I've been mostly doing what I call maintenance runs, which is when I already downloaded a tag a while back but now I'm running it again to grab the new stuff as well as older posts that have had the tag added to them.
It really depends how far I make it, but something I noticed is that it aligns more or less to when I would have received a 429 notice. With my usual settings I would get 429s at a certain frequency, so it's easy to see in the output.
I followed the usual steps to get around Cloudflare (#4844) but it's still happening. Doesn't help that https://api.rule34.xxx/ doesn't list rate limits so shooting in the dark, I'm currently increasing the sleep times in a trial and error fashion, but I wonder if there's anything else that could be done, it's currently my biggest bottleneck among the 14 boorus I download regularly (my script downloads the same tag from all boorus).
Something I notice when the Cloudflare error appears is that when opening the post URL that triggered it in the browser, most times I don't see a CF challenge to solve, and the post loads as expected. A few times a CF challenge appeared but it autosolved itself and redirected me to the post immediately.
Current relevant settings:
{ "rule34": { "skip": true, "tags": true, "notes": true, "sleep": 2.0, "sleep-request": 15.0, "sleep-429": 500, "cookies": "~/.config/gallery-dl/cookies/rule34.txt", "cookies-update": true, "user-id": "xxxxxx", "api-key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" } }