Skip to content

Introduction to Sync Gateway Accelerator

Jump to bottom
James Nocentini edited this page Sep 21, 2016 · 4 revisions

Overview

Sync Gateway Accelerator provides enhanced scalability for Sync Gateway. In addition to having Sync Gateway nodes, you can now deploy Sync Gateway Accelerator nodes to handle the server mutation listener and channel caching work which will free up resources on the Sync Gateway nodes for connected clients.

All clients should continue to communicate directly with the Sync Gateway node(s). There are no changes required at the application level (Couchbase Lite, Web).

Why should you consider using SG Accel?

Sync Gateway can be scaled horizontally (adding extra nodes to your Sync Gateway cluster) to handle large amounts of read traffic. However, there's a limit to how much write traffic a standard Sync Gateway cluster can handle.

To support client replication (specifically the changes feed), Sync Gateway needs to monitor changes happening in the backing Couchbase Server bucket, apply security filtering, and stream those changes to users. To optimize this processing, Sync Gateway maintains an in-memory cache of recent changes in each channel, and uses that cache to serve _changes requests. However, each Sync Gateway node in the cluster needs to monitor the entire server mutation stream - as write throughput increases, each node needs to do more work to manage its cache.

Sync Gateway accel enables greater scalability by distributing the stream processing work across a cluster of sg-accel nodes. A cluster of sg-accel nodes work the server's mutation stream, and persist the channel information to a channel bucket. The cluster of Sync Gateway nodes are only responsible for serving client requests via the REST API. This makes it possible to scale both sg-accel and Sync Gateway horizontally to handle much larger write throughput.

Basic Setup

Setting up your Couchbase Server

Define a new bucket in your Couchbase Server - this is the bucket Sync Gateway accel will use to store channel information. These instructions will refer to this bucket as the "channel bucket", and the regular Sync Gateway bucket as the "data bucket". The sizing requirements for the channel bucket will vary depending on the size of your data bucket. (For 1.4 we'll be able to provide more detailed sizing recommendations)

Setting up your Sync Gateway Accelerator cluster

  1. Download and install sg_accel on the nodes in your Sync Gateway Accelerator cluster. You can download Sync Gateway from the Add-ons section or using rpm. See the Sync Gateway installation guide.
  2. Next, you can specify the location of the channel bucket and of the data bucket in the Sync Gateway accel configuration file.
{
  "log": ["HTTP+"],
  "adminInterface": "127.0.0.1:4985",
  "interface": "0.0.0.0:4984",
  "cluster_config":{
    "server":"http://localhost:8091",
    "bucket":"default",
    "data_dir":"."
  },
  "databases": {
    "default": {
      "server": "http://localhost:8091",
      "bucket": "default",
      "channel_index": {
        "writer": true,
        "server":"http://localhost:8091",
        "bucket":"channel_bucket"
      }
    }
  }
}

Some items to note:

  • The cluster_config section is required to manage communication between your sg-accel nodes. The server property is your Couchbase Server address, and bucket is the name of your data bucket.
  • The default database is the standard Sync Gateway database config, and the server and bucket should target your data bucket. The channel_index section specifies your channel bucket connection information.
  1. Start sg_accel on each node from the command line.
sg_accel /path/to/sg-accel-config.json

Setting up your sync_gateway cluster

  1. Install sync_gateway on the nodes in your sync_gateway cluster.
  2. Create a new file called sync-gateway-config.json to hold the configuration for Sync Gateway. On each node, customize your database config to add the channel_index property.
{
  "log": ["HTTP+"],
  "adminInterface": "127.0.0.1:4985",
  "interface": "0.0.0.0:4984",
  "databases": {
    "default": {
      "server": "http://localhost:8091",
      "bucket": "default",
      "channel_index": {
        "server":"http://localhost:8091",
        "bucket":"channel_bucket"
      }
    }
  }
}
  1. Start sg_accel on each node from the command line
sync_gateway /path/to/sync-gateway-config.json
Clone this wiki locally