Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Persistent Volume and changing IPs #82
In the "Best Practices" it states:
Unfortunately this won't work when recreating the Docker container with a new IP, since the IP addresses of hosts is part of this persistent volume. Since this scenario is a very usual case when working with lots of containers in practice, I would like to ask how the official guide from couchbase is, in order to avoid breaking couchbase instances when e.g. updating (usually done by recreating containers) or migrating (usually done by removing and creating on different location) them.
Thank you in advance.
referenced this issue
Nov 8, 2017
I've found on AWS that the best approach is to add nodes to the cluster by internal IP address and not by name or external IP. We do not expose Couchbase server nodes directly to the public Internet, but by other services that use Couchbase as a back end database.
AWS guarantees not to change your internal IP address when you stop/start or restart nodes which makes this approach viable. If you absolutely need to expose couchbase across the internet then attach an elastic IP to your instance and use that.
There are I sure other approaches, but this works well for us.
@hookenz are we still talking about containers?
Anyway, not changing the (internal) IP is obviously one solution, yet it feels like a dirty work around and may not work in many cases (think about a very busy cluster where private IPs are assigned every few minutes / seconds to containers, which come and go regularly).
To be honest, I am personally dreaming of a couchbase cluster where each node takes care of that, without having an additional operator, which may fail (SPOF), has to be set up first (requires orchestrating / wiring), and so on. Not having router/operator/management node types (like found in MySQL Cluster, MongoDB, ...) made couchbase very attractive.
@cha87de I do agree with you on that, having Couchbase doing the scale for you would be wonderful. I think Couchbase is not far away from having that happening, it already takes care of a lot and it handles failover clustering very well together with the Cluster-API. But yeah, the Change-IP problem is a bit of a pain and an obstacle on the road to achieve a seamless scale.
We are simply using the docker-engine (docker-compose-service) directly, no Kubernetes or AWS and it works only in a non cluster mode.
For posterity, we've managed to work-around the scenario of changing container IPs on AWS ECS container instances using
Our ECS infrastructure is built entirely using CloudFormation and updates to the tasks/services always necessitate container re-creation, which changes their IPs and effectively destroys any CB clusters running on them. Our ECS container instances are part of an ASG with dynamic Lambda function scaling them up and down based on demand for resources.
We always set the first node in the CB cluster to use a CFN generated private namespace DNS name, which is resolvable by all hosts in the VPC. We also employ a "helper" container and shared EFS storage between all nodes, where "state" is kept. The state includes the current cluster IPs, the shared
The CB containers run a bootstrap script on startup and using state information update
It's not an ideal solution, since the bootstrap and the helper scripts are big enough to be brittle, but it seems to work in our case. We can scale our ASG from 1 to X nodes and back down without losing data.
We use scale-in protection to make sure the first "master" CB node is never destroyed as a result of the ASG terminating the container instance hosting it. We also protect the node by disabling EC2 API termination.
So in summary, this it is possible to achieve, but it does require quite a bit of wiring: