[FEATURE] Dealing with 429's?

[baccccccc]

I apologize for marking it as a "feature" because it's more like a discussion topic. Or maybe it sounds like a proposal for a wiki page.

HTTP code 429 is basically DDoS-Guard, right?

What are the typical ways to deal with it?

I have a feeling that it worked slightly better in v4 than in v5. But I have no data to back up this claim with.

I set my rate limiting options to one concurrent download, and I am still hitting a bunch of 429s on bunkr.

Rate_Limiting_Options:
  connection_timeout: 15
  download_attempts: 10
  download_delay: 1
  max_simultaneous_downloads: 1
  max_simultaneous_downloads_per_domain: 1
  rate_limit: 50
  read_timeout: 300

Every time I run CDL to download a significantly large forum thread, there's still a bunch of failed downloads with 429. Previously, in v4, all it took was re-launching CDL again (and maybe once again) to download the remaining file. But not anymore. I tried launching CDL four or five times in a row, and there's still about a dozen files being blocked.

I confirm when using the same link in a browser, download starts. So, it's not like it's actually a 404 or 451 disguised as 429 as it sometimes happens with other downloads.

So maybe I'm just getting unlucky, or v5 is somehow more "efficient" (aggressive?) in trying to download stuff from bunkr? Any further tuning recommendations?

[Jules-WinnfieldX]

So coincidentally 429's got a lot worse right as I released V5. In testing I basically didn't run into them, now they are rampant. I feel like it is variable server side depending on server load, and I'll never be able to tune the program for it.

[baccccccc]

just thinking out loudly... what if you built a random delay (think, 1..5 seconds) between retry attempts? Can it, at least in theory, help to some degree?

[Jules-WinnfieldX]

I doubt it. There already is a 1 second delay between downloads.

This likely will be something I have to play with, but there is not going to be getting rid of it unfortunately.

[Jules-WinnfieldX]

1 second for bunkr*

[baccccccc]

is it hardcoded? What if I want to set it to two or three seconds and see if it helps in my case?

[Jules-WinnfieldX]

oh. I think I found it.

[Jules-WinnfieldX]

so. The arg I was going to suggest you use, I never actually use in the code. Love that. But I also found a typo meaning bunkrr downloads / requests weren't following the rate limits I set for them at all

[Jules-WinnfieldX]

Give me a second to fix some shit surrounding this that I'm finding

[Jules-WinnfieldX]

5.0.49 is going up now, see if it helps

[kurfuu]

Will this update be put into the Cyberdrop-DL V5 Start Files at somepoint.
im no good at navigating github, and cant seem to figure out how to add the updated files into the Start Files folder i currently have. <3

[Jules-WinnfieldX]

Will this update be put into the Cyberdrop-DL V5 Start Files at somepoint. im no good at navigating github, and cant seem to figure out how to add the updated files into the Start Files folder i currently have. <3

The start files will automatically install updates such as this one I just put out, yes. You don't need to do anything on your part besides maybe running the start file once or twice for the update to install.

[baccccccc]

wait, what are the start files? 👀

[Jules-WinnfieldX]

wait, what are the start files? 👀

https://github.com/Jules-WinnfieldX/CyberDropDownloader/releases/tag/Release

the 5.1 zip contains them. You'd have to severely edit them for them to work with the windows store python install