Allow server healthchecks to run concurrently

[mbStavola]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you expect to see?

While reading through the source I was surprised to see that when Traefik performs a healthcheck for servers within a service, it performs them sequentially. I whipped up a quick test and confirmed that this is actually how it behaves. Interestingly, this isn't the case for services which seem to get launched as separate goroutines.

This means that, in the worst-case scenario, the last server in the service's pool will get healthchecked frequency + (timeout * poolSize) seconds apart. In my opinion, this makes the healthcheck frequency config a bit misleading as I would expect that each server is checked <frequency> seconds from the last time it was checked.

What I would like to see is an option to change this behavior and run the healthchecks concurrently. This way, each server in the pool is healthchecked roughly every X seconds defined by the frequency parameter, regardless of the (mis)behavior of other servers in the pool. The config could be something like this:

traefik.http.services.my-service.loadbalancer.healthcheck.concurrent=true

[AhmedGrati]

I can take it if there's no one who is working on it.

[ldez]

We flagged this issue as a proposal, which means that we are expecting use-cases and feedback before deciding on the topic.

[emilevauge]

Hi @mbStavola,
We think it would be a great addition to the product!
However we don't see a value making this configurable, it should be concurrent or not.
Additionally, we see a possible impact on the load as all healthchecks would be done at the same time. It would be a good idea to create a pool of healthchecks with a limited size (configurable) to throttle the healthchecks pipeline if needed.
If anyone in the community wants to tackle this yopic, we would love to help!

[mbStavola]

@emilevauge My primary reason for making it configurable was to preserve the current behavior, as I'm not sure if anyone is relying on it (as you mention, load may be a factor). But your proposal for configurable pool size makes a ton of sense as someone who wants to preserve the behavior can just set the pool size to 1. I like it!

Personally I would love to work on this, but I don't think I have the time anymore. Hopefully someone else can step up and look into it.