HTTP response 429 (Rate limiting)

[forky2]

I get a lot more "Retrying pull page" messages than I used to. I turned on --debug messages and I can see many HTTP 429 responses which indicates that rate limiting is being applied.

I've not yet investigated whether this could be caused by a web proxy in my organisation, so I just thought I'd query if anyone else reports the same. If it's a problem, I'll look to implement an optional rate limiting parameter.

[forky2]

I tested from home (with no web proxy implications) and I had the same problem. The marketplace is throwing status 429 partway through a sync. I've made a modification to handle this scenario with a backoff timer and it seems to be a suitable solution. It will be tested over the weekend and I'll submit a PR once I'm sure it's a good fix.

[Christian-Johansson]

any updates on your fix there @forky2 ? did it work as aspected looking for a solution like this but there are a few repos that have similar functionality

[forky2]

Yes, sorry I've been heavily distracted. I'll submit a PR today.

[forky2]

@Christian-Johansson Feel free to try my change.

[LOLINTERNETZ]

Thanks for pulling this together!

I just had a quick look at the PR and it looks good. I might wait a day or two before merging(?)

I'm kind of surprised how good of a run we've had without hitting rate limits. Out of interest, what rough order of magnitude is your connection? It's not the most parallel code, so testing from Australia can somewhat mask rate limits for serial calls.

[forky2]

No problem at all. I agree, we've had a good run. :) My connection is ~80Mbps. Not exactly fast. Time of day could be a consideration.

[Christian-Johansson]

I've tested it now and it works flawless no more e429 when getting over 10k extensions to sync