Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Operator reconfiguration / storage migration on the fly #435

Open
alaric-dotmesh opened this issue Jun 6, 2018 · 0 comments
Open

Operator reconfiguration / storage migration on the fly #435

alaric-dotmesh opened this issue Jun 6, 2018 · 0 comments
Labels

Comments

@alaric-dotmesh
Copy link
Contributor

@alaric-dotmesh alaric-dotmesh commented Jun 6, 2018

It's a pain to change the configuration of a running Dotmesh k8s cluster (especially changing storageMode). We could automate this process:

  • Make the Operator watch for ConfigMap changes and reload the configuration when they happen.
  • Make the Operator notice, as part of its "find dotmesh pods that must die" logic, pods that are configured in the wrong way (put an annotation on the pods containing the bits of the configuration that affect them, so it can be checked).
  • However, rather than just killing these pods, ask them to terminate "softly"; that means waiting for all containers with mounts on that node to finish (and not allowing any new ones to start in the meantime), then ensuring that the pod is not Master for any dots, before terminating. When it starts afresh with new storage, it'll re-replicate everything and get up to speed.

Soft termination mode will be useful for controlled shrinkage of clusters, too.

@alaric-dotmesh alaric-dotmesh added the task label Jun 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.