Sequence Handling when using a Distributed Index
The existing (pre-1.2) Sync Gateway architecture uses a single global integer sequence value to provide the ordering guarantee needed for replication. At a high-level - every time a Sync Gateway node writes a new document to its bucket, it allocates a sequence for that document by incrementing a global sequence document. More details on the current approach can be found in Sequence Handling.
To support the scalability goals for 1.2, we needed to move to a new sequence model, to address the following limitations of the current approach:
- contention on the global sequence document as write load increases
- sharding the Couchbase Server mutation feed means that each Sync Gateway node only sees a subset of the mutation feed. This makes it much more difficult to buffer sequence numbers and identify the stable sequence for the cluster.
Couchbase Server doesn't maintain a single global sequence. It's the nature of a distributed database that maintaining such a sequence is impractical. However, each Couchbase Server vbucket maintains an internal sequence value that's guaranteed to be monotonically increasing.
A vector clock made up of all the {vbucket, vbucket sequence} values for the cluster provides the information and guarantees that Sync Gateway needs for replication. When using Sync Gateway with a distributed index in 1.2, a vector clock made up of vbucket sequences is used to manage replication.
The good news is that the full vector clock provides the information we need for replication. If a client sends us a vector clock as a since value, for instance, we know that client has been sent all the appropriate documents for each vbucket up to the corresponding vbucket sequence in the clock.
The bad news is that the vector clock is too large to send back and forth to clients as the full clock - the standard Couchbase Server deployment has 1024 vbuckets, and each vbucket sequence is a 64 bit integer. Even compressed, that's too large to be sending with every entry on a changes feed.
To address this, each sequence is hashed and stored in a persistent lookup table. The hashed value is sent to the client, and on subsequent client replications Sync Gateway converts the hashed value back to the full clock.
When using a distributed index, sequence formats have a somewhat different format than the current format. As always, sequence values should be treated as opaque by clients. However, understanding sequence formats can help debug and diagnose issues. Here are the sequence formats that may be seen in 1.2. The first four entries describe the pre-1.2 functionality (and may still be seen post-1.2 when running Sync Gateway without a distributed channel index.
| Format | Index Type | Description |
|---|---|---|
| 123 | Non-distributed | Basic global integer sequence |
| 123:100 | Non-distributed | Global integer sequence, doing channel backfill. In this example, Sync Gateway is sending sequence 100 as part of a channel backfill that was triggered by sequence 123. |
| 90::100 | Non-distributed | Global integer sequence, with handling for delayed sequences. In this example, Sync Gateway is sending sequence 100, but only guarantees completeness to sequence 90. |
| 90:123:100 | Non-distributed | Global integer sequence, with both delayed sequence handling and channel backfill. In this example, Sync Gateway is sending sequence 100, as part of a backfill triggered by sequence 123, but only guarantees completeness to sequence 90. |
| 200-0 | Distributed | Basic vector clock sequence. In this example, 200 is the hashed sequence value, and 0 is the hash collision index |
| 50.123 | Distributed | Vbucket sequence format - used for some intermediary sequence values that aren't needed for subsequent client requests. These are in the format vbucket.sequence - so in this example, the entry represents sequence 123 from vbucket 50 |
| 200-0:50.123 | Distributed | Vector clock sequences, doing channel backfill. In this example, Sync Gateway is sending sequence 123 from vbucket 50, during a channel backfill triggered by the vector clock hashed to 200-0 |
As mentioned above, this change to the sequence format is designed to be transparent to clients. Sequences have always been opaque to clients by design, so there aren't any client changes needed to support the new sequence formats. However, there has been significant refactoring on the Sync Gateway side to support vector clock sequences - particularly around _changes feed processing.
The sequence hashes are stored in a Couchbase bucket, to ensure they are available across the Sync Gateway cluster. By default, they are stored in the same bucket used to store the channel index. However, users can opt to specify a separate bucket to be used strictly for hash management in the Sync Gateway config.