Q: Servers, replicas, groups, shards, and tablets. Do I understand this correctly?

[dzrw]

Can you check my understanding of the relationships between servers, replicas, groups, shards, and tablets described in the documentation?

1. A cluster is the entire deployment of a dgraph instance. A cluster contains at least one server, at least one zero, and zero or more RAFT quorums.
2. A zero is a super node. It monitors and balances the data held on servers.
3. A server is a worker node within a cluster. It contains the actual data on disk, and serves both reads and writes. A server contains exactly one group.
4. A group is a logical collection of predicates. A group contains at least one predicate.
5. A predicate is a logical collection of ground truths.
6. A tablet is a physical collection of ground truths stored on the disk. A tablet contains the physical data for one predicate.
7. The terms shard, data shard, and tablet as used in the documentation are synonyms.
8. A replica is a server that belongs to a distinct quorum. Depending on the value of the --replicas flag all servers are replicas (>=2) or none of them are (==1).
9. A quorum consists of a set of replicas that provide readwrite access to consistent copies of the exact same group where consensus is handled by RAFT. A quorum contains a least two servers, but ideally 3 or 5 to avoid split brain situations.

Capacity Planning

The --replicas tunable applies to the entire cluster and not to a particular predicate. To plan the number of servers for a cluster with replicas > 1, you have to be able to predict the number and size of predicates for your schema, and then solve a bin packing problem to estimate how many groups dgraph would optimally want to break that into.

When sizing a cluster, if you opt for few groups and large disks, you'll run into throughput issues on reads due to page faults. Alternately, if you opt for more groups, then you'll have to provision redundant servers for predicates that see low query traffic resulting in higher TCO.

If I understand things correctly, maybe the thing to do here is to let folks set replica policy on a per predicate basis, and then let the zeroes migrate tablets based on policy rather than cluster-wide settings?

[dzrw]

In the absence of fine-grained tunables, it seems like the thing to do is to provision 5 servers and set --replicas 5, then pick the largest, fastest SSDs you can. That way, you pay the price of RAFT consensus once, and can avoid the group/predicate optimization problem. Write performance no longer scales as adding more than 5 nodes to a RAFT quorum isn't recommended (see the paper). Obviously, storage won't scale either as you're locked into the size of the disk you chose. And, of course, reads don't scale when you pin the size of your cluster.

Well, you could choose 1, 3, or 5 servers, and kind of have small (1), medium (3), and large (5) deployment options (taking a maintenance window to upgrade as the dataset grows).

[pawanrawal]

1. A cluster is the entire deployment of a dgraph instance. A cluster contains at least one server, at least one zero, and zero or more RAFT quorums.

Sounds right.

2. A zero is a super node. It monitors and balances the data held on servers.

Yes, right. Apart from rebalancing the data, it is also used for assigning UIDs and transaction timestamps.

3. A server is a worker node within a cluster. It contains the actual data on disk, and serves both reads and writes. A server contains exactly one group.

Right, though it might be more appropriate to say that a server serves data for only one group.

4. A group is a logical collection of predicates. A group contains at least one predicate.

Right. Though initially, a group may not contain any predicate.

5. A predicate is a logical collection of ground truths.

A predicate is just the middle part in a <Subject> <Predicate> <Object>/ObjectValue . RDF NQuad. Dgraph shards data by a predicate. Therefore all data for a predicate must reside on the same server.

6. A tablet is a physical collection of ground truths stored on the disk. A tablet contains the physical data for one predicate.

A tablet consists of all the data corresponding to a predicate, including all the indexes.

7. The terms shard, data shard, and tablet as used in the documentation are synonyms.

Sounds correct.

8. A replica is a server that belongs to a distinct quorum. Depending on the value of the --replicas flag all servers are replicas (>=2) or none of them are (==1).

Sounds correct. Though note if replicas had the value 2 and you just brought up three nodes, then the third node would be the only server serving group 2. So it's not a replica.

9. A quorum consists of a set of replicas that provide readwrite access to consistent copies of the exact same group where consensus is handled by RAFT. A quorum contains a least two servers, but ideally 3 or 5 to avoid split brain situations.

I think the docs are a bit confusing with regards to a quorum. Quorum is the minimum number of nodes that must be present for writes to go through which is n/2 + 1 if a group has n servers.

Yeah, the --replicas applies to the entire cluster.

To plan the number of servers for a cluster with replicas > 1, you have to be able to predict the number and size of predicates for your schema, and then solve a bin packing problem to estimate how many groups dgraph would optimally want to break that into.

Since Dgraph moves tablets around different groups based on their size at regular intervals, you don't have to solve the bin packing problem.

If I understand things correctly, maybe the thing to do here is to let folks set replica policy on a per predicate basis, and then let the zeroes migrate tablets based on policy rather than cluster-wide settings?

That sounds very complicated for the user. Since Dgraph tries to ensure that different groups serve similar sized data, that should be good enough.

[zero-master]

Thanks for the details.

Can you please also describe how the data is rebalanced?

If not, a few links would immensely help.

[dzrw]

@pawanrawal Thanks for the review. I'm still not sure how I should go about sizing a cluster of EC2 nodes without more fine-grained tools for managing replicas, but your other answers were helpful.

[rahst12]

"A predicate is just the middle part in a /ObjectValue . RDF NQuad. Dgraph shards data by a predicate. Therefore all data for a predicate must reside on the same server. "

If you were modeling Twitter followers, several hops out - wouldn't the predicate "follows" eventually fill the server?