-
Notifications
You must be signed in to change notification settings - Fork 880
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docker should avoid re-using IPs too quickly #1945
Comments
There is a PR open that aims to allocate IPs sequentially: #1788 Regarding in a microservice application you would like to spawn different mysql clusters as different services, and you will use DNS in the application to always point to the correct instance. Also on a single network there should never be use of the same IP for 2 running containers. Closing this one, feel free to follow up if you have further questions |
Thanks @fcrisciani that PR looks like it will solve this!
Just to follow up and clarify the problems... This is already what we do, but collisions can still happen relatively easily. We're running hundreds of MySQL clusters on Kubernetes among other data services. Ingress traffic from clients is generally never the problem because it comes through virtual service IPs, the problem is generally with direct communication between replicas. Things like redis-sentinel, zookeeper, hbase, kafka, and etcd all require direct container-to-container communication within their clusters. We've seen independent redis-sentinel clusters accidentally discover each other and "merge", or MySQL hosts accidentally trying to replicate from a different MySQL cluster, etc. Each of these individual cases has other solutions, eg: ensuring every cluster of a given service has different auth credentials so connections fail, etc; but avoiding rapid IP reuse is a simple change that really avoids a lot of edge cases in production so this is a very welcome change and will probably reduce pain for a lot of people. Thanks! |
sure @PaulFurtado no problem, thanks for clarifying the case. This commit mitigate for sure the condition but still when you will get close to finish the address pool you will still experience situation like this. |
When allocating an IP for a container, libnetwork always chooses the lowest IP. This can be a big problem when a container exits and a different container running the same type of service starts on the same IP. Ex:
This is just one example, but this can happen for just about any service, especially HTTP services in a microservice environment due to their quantity.
The text was updated successfully, but these errors were encountered: