Thin arbiter quorum for 2-way replication

[pranithk]

Thin-arbiter design document details the design which introduces new kind of quorum in 2-way replication in AFR which adds quorum at a brick level. One of the use cases this can be used in is stretch cluster scenario. Users can also use it if they find it fits their use case better than arbiter. Once this becomes stable, we should start looking at deprecating 2-way replication IMHO.

Please add your comments to the document itself, so that the complete discussion is in the document.

[slesru]

For me it is absolutely not clear how this can deprecate 2 way replications- we use just 2 machines in our cluster with it.

[ShyamsundarR]

@pranithk What are the implication when upgrading a cluster, with this approach? Today, the upgrade process, is to upgrade one of the replica bricks and then wait for healing to catch up before upgrading the other brick as in here.

So with the thin arbiter, does this change at all? Or, how does the upgrade procedure change (including upgrading the arbiter in future releases)?

[pranithk]

@slesru sorry for the delay in response, was on vacation for a while.
2-way replication leads to split-brains. There is no way to avoid it. The only way to avoid it is if we have at least 3 nodes. So we have 3 variants now to avoid split-brains.

1. 3-way replication
2. arbiter
3. thin-arbiter

[pranithk]

@ShyamsundarR upgrade procedure is same. At least I am not able to see any conflicting cases to prevent it at the moment.

[amarts]

Some discussions I had with team offline, which I am recording here for wider audience.

Challenges of management layer latency issues

One of the main goal of thin arbiter is to have 3rd node in remote/cloud setup which can have higher latency. But our management layer would become very slow with this, if we want to manage the thin-arbiter through glusterd (or gd2) process.

• Possible solution: Provide a separate command which will always run, say gluster-tiebreaker, which can be a script which will run glusterfs itself as client process (which can connect to one or more glusterd processes with backup-volfile server).

Now, the volume create can add this arbiter node as a common option, which will make sure the same will be added to all the replica pairs.

A clear document with all tested steps to revert a bad brick to good brick in the thin arbiter.

This is very important in this design as there are possibilities where just because of one (or few) file pending heal, a whole brick, which may contain millions of files may be marked as bad. In that case, we need a mechanism to override the code-path by resetting the relevant flags.

---

Feedback welcome @pranithk @itisravi @karthik-us @aspandey

[gluster-ant]

A patch https://review.gluster.org/19545 has been posted that references this issue.
Commit message: cluster/afr: Implement thin-arbiter translator

[gluster-ant]

A patch https://review.gluster.org/19835 has been posted that references this issue.
Commit message: afr: thin arbiter changes

[pranithk]

I found a race bug in the earlier design where split-brains can happen. I just had a discussion with @aspandey @itisravi @karthik-us about the problem and solution. Ashish will be updating the design doc with the relevant changes. This is going to be fixed as a separate bug after the initial implementation.

@aspandey Please add a comment once you update the design document.

[aspandey]

I have updated the design document as per the discussion on issue raised because of race between shd up call and transaction.
Please review it and provide your comments.

[pranithk]

@amarts @ShyamsundarR There was no consensus in the document which brought doc/spec approved flags about the format of the document last I checked. What do you guys think we should do to get the spec/doc approved flags considering that the google-doc already has the documentation of the work that is done? Should we send one more patch to glusterfs-specs?

[gluster-ant]

A patch https://review.gluster.org/19940 has been posted that references this issue.
Commit message: cluster/afr: shd changes for thin arbiter.

[amarts]

Flags provided after revisiting the document and the CLI format discussions in glusterd2 issues (referenced above).

[hsafe]

@pranithk
Can you please explain a bit when you say: 2-way replication leads to split-brains ?
I am trying to understand if I run a 2x replica nodes each one brick ...in case of one fails and come back live, if I have a fallback config on the clients the two nodes can heal and sync each other or not? Can you pelase clarify?

[pranithk]

@hsafe For the case you explain, it will be fine.
Let us say the setup has 2 bricks b0, b1
Consider the following case:

1. File 'a' is created when both the bricks b0 and b1 are available.
2. Now when a brick b0 goes down, 'a' is added with content 'abc'
3. Now b0 comes up, but before heal/sync can happen b1 goes down.
4. If you add some other content say 'def' to 'a', it will succeed.
5. Now when b1 comes back up. You have 'a' on b0 with 'abc' and 'a' on b1 with 'def' both of the bricks think they have the correct copy leading to split-brain situation.

[hsafe]

@pranithk
Thanks that explained everything...:)