Time between requests to prevent Hetzner Rate Limit

[maxfriedmann]

At first: Many thanks for this great docker machine driver.

We've been using it for a week in testing, without any issues. Today we started using it in production and ran into the rate limits of hetzner itself. It seems like if you once hit the rate limit, you'll never get out of that situation since the machine driver keeps on retrying all requests in an endless loop.

Would it be possible to implement a cool down time when 429 errors happened?

Unfortunately I am not a Go Programmer :/

[maxfriedmann]

The actual error from the driver is
ERROR: Error creating machine: Error in driver during machine creation: could not create server: we are unable to provision servers for this location, try with a different location or try later (resource_unavailable) driver=hetzner

I am not sure if the rate limiting is the consequence of the driver retrying to create machines or vice versa.

[JonasProgrammer]

Hi,

first of all, thank you for bringing this up. Could you perhaps describe the process that got you into the rate limit a little bit more?

While the 429 error absolutely makes sense, the message seems a bit odd to me: It reads more like a resource exhaustion in the given region, rather than you 'spamming' the cloud API.

If it is due to an external reason, IMO the fail-fast option (i.e. the current behaviour) is the best way. On the other hand, if it comes down to requests being sent too frequently the driver could indeed slow itself down.

I'll have to look, whether the status code is actually exposed in the error interface returned by the hcloud library; if not, I don't really see a way of handling this gracefully.

If it is available, would it be ok for your use-case to make this an explicit opt-in as a command line option? The driver is being used in automated scenarios as well and I think for the non-manual case, failing fast is preferrable to waiting an unspecified time.

Further discussion welcome.

[maxfriedmann]

I totally agree. The Hetzner Support took quite long to let me know that their cloud actually ran out of resources. Me trying to boot new instances led to the 429 errors which are connected to the account you're using, not an api key or login. So my whole team suddenly got the 429ers.

It was a combination of low resources and, afterwards, my CLI tool going wild. So I would agree if you'd say that it was an unusal situation. I'd treat it as super low priority if I would be you.

If it takes 5 minutes to implement an additional --time-between-calls parameter, then I'd be very happy. Otherwise I just keep my fingers cross that hetzner now has enough resources in all regions.

[maxfriedmann]

Unfortunately we had to move back to root servers since the performance of the cloud servers wasn't satisfying. Thanks for your support!

[JonasProgrammer]

Unfortunately we had to move back to root servers since the performance of the cloud servers wasn't satisfying. Thanks for your support!

Sorry to hear that but thank you for taking the time to come back and close this. Perhaps this could still be implemented in the future, but wrapping every operation in a retry-loop (and maintaining a consistent internal state) turned out to be harder than I initially anticipated, especially with cleanup stuff. I will keep a mental note, but with your initial issue being an external problem, I don't think the benefit outweighs the additional complexity brought in.

Still glad, you found something working for you in the end, even if it turned out to be the good old bare metal.