Improve Dgraph upgrade experience by supporting in-place & rolling upgrades

[sleto-it]

Experience Report

This is a feature request to add an additional upgrade method, the in-place upgrade, which can coexist with our current logical upgrade method

When the in-place upgrade will be available, this is a FR to support rolling upgrades (one node at a time) of Dgraph clusters, so that the upgrade can be done while the cluster remains up and running, and ideally while it keeps serving reads and/or reads and writes

What you wanted to do

• Upgrade in an easy way a running Dgraph cluster, especially when upgrading to a new patch release (e.g. from 1.1.0 to 1.1.1)
• Minimize operational efforts
• Minimize downtime

What you actually did

The upgrade is currently possible, but it always requires an export and import (logical upgrade)

Why that wasn't great, with examples

While it is understandable and acceptable that sometime a logical upgrade might be required (e.g. due to some incompatible changes, to re-shard the cluster, etc), it is important to improve the upgrade experience as much as possible e.g. by introducing a new upgrade method: the in-place upgrade, which would allow the installation of the new package/version and the reuse of the current data directory. The upgrade process can be responsible to run some in-place upgrade operations in the existing data directory, when needed

In-place upgrade will then hopefully also allow rolling upgrades, where we upgrade one node at a time, the cluster remaining up and operational in the meantime

The problem related to the current upgrade approach is that, in addition to the export and import, it requires to set up a new cluster, if you want to minimize downtime (blue-green deployment for upgrades) - which, from an operational point of view, is costly in terms of required time and efforts

Any external references to support your case

• other database vendors provide this feature
• increasing requests from our user base - examples: link1, link2
• doc page on the upgrade, e.g. from 1.1.0 to 1.1.1

[marvin-hansen]

Upvote, 100%. This is critical for operations.

A preliminary way to deliver rolling updates would be a reference process in Kubernetes by setting up a second deployment and then re-route traffic to the updated version using the service.

https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

Obviously, not all deployments run on k8s but writing a tutorial about rolling updates in Kubernetes would already make things way better and fill the time until a proper solution is there.

If possible, please update the documentation!

[artooro]

A side effect of not having a way to upgrade Dgraph without or at least with minimal downtime will be that users just don't upgrade, or rarely upgrade.

@marvin-hansen I've thought about finding a way to spin up a new deployment, and switch the kubernetes service to it, but haven't figured out a good mechanism to keep the existing deployment up in a read-only mode, as there will be the period of time where you have to export, and import to bring the new deployment up.

[marvin-hansen]

@artooro

Have you looked into switching traffic through ingress routing to reduce deployment risk?
https://konghq.com/blog/reducing-deployment-risk-canary-releases-blue-green-deployments-kong/

Not perfect, but it seems practical compared to service switching.

where you have to export, and import to bring the new deployment up.

Back in the days, some Linux admins used to solve that with dsh, the “Distributed Shell” by piping the export stream from one box directly into the import of another box. As long as you stay within the same K8s cluster, this might do the trick relatively fast.

read-only mode

100%, much needed. Would you fill a separate issue?

I am actually having the case that I can't upgrade much after staging and it's already causing quite some operational headache. I guess that's what you get when you adopt early tech.
On the other hand, my experience comes down to, when it runs, it actually runs well.

[campoy]

Hi there,

This is totally part of our roadmap, and just in case people missed that @sleto-it is part of our Customer Success team.

Additionally, we recently improved our helm charts to help with initial deployments.
Finally, we are working on having a Kubernetes operator too, which should help with these upgrades.

All of those things will be coming up during 2020, and if you have some specific demands this is a great place to leave your feedback, as it will be useful to prioritize the many ways we can improve your experience.

Thanks for all of those that already did that on this issue, btw! You rock!

[artooro]

@marvin-hansen yes I'm aware of the various deployment methods. Databases are a bit different than stateless web apps as well. :)

I'm excited to see eyeballs on this type of capability. Also glad to see @campoy part of the team.

[sleto-it]

Hi all,

As always, thanks for sharing your feedback and experience

Generally speaking, regarding the read-only mode, what you can do is to restart all your alpha instances using the option --mutations disallow. You will need to add this option and restart all your alphas if you want the entire cluster to be in read-only mode

Depending on how you deployed Dgraph, e.g. manual start / systemd / k8s etc, the specific way to set this option can be different

Option ref: https://docs.dgraph.io/master/deploy/#restrict-mutation-operations

Hope this helps

Thanks,

[artooro]

Great tip @sleto-it, I will be using that method on my next upgrade, as our live backend only does read-only transactions, while an event driven micro service handles all the writes (mutations). So if the events backlog while the upgrade is happening that's perfectly acceptable in my case as longs as reads continue to work.

[marvin-hansen]

Is there some progress on this issue?

I'm trying to upgrade right now, and, quite frankly, the process doesn't even make any sense.

Is there a walk-trough the process because, in its current state, the documentation leaves out so many important details and raises so many questions. I filled yet another issue, but I find it astonishing that something as basic as import/export has been made so complicated.

As a preliminary workaround, can at least the documentation get some improvement so that one can do an export / import within ~20 minutes for a small DB?

[minhaj-shakeel]

Github issues have been deprecated.
This issue has been moved to discuss. You can follow the conversation there and also subscribe to updates by changing your notification preferences.