Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Kubernetes killed an alpha pod in the middle of retrieving snapshot #2698
If you suspect this could be a bug, follow the template.
Server 3 lost its leadership, then was considered to be behind the group, and was getting a snapshot. This is when k8s killed it, packed up its contents and shipped it to another container.
It is not clear what we could have done at Dgraph level to avoid this, given our lack of control over a forceful shutdown and a pod move. Therefore, what we want to determine here is if kubernetes is a good choice to run Dgraph cluster. Can we find a way where we can ensure that k8s would allow the pods to shutdown gracefully? Or, should we recommend users to not use k8s for Dgraph.
One possible way is to increase the graceful shutdown to a sufficiently large duration, like say 10 minutes.