Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
bin
integration/token_syncer
resources
src/token_syncer
test/token_syncer
.gitignore
README.md
project.clj

README.md

The token-syncer is a command-line application designed specifically for syncing tokens across Waiter clusters. This application can be set up as a cron job to sync tokens on independently running Waiter clusters in different datacenters. Syncing tokens across datacenters facilitates the hosting of highly available services across multiple Waiter clusters. All that is needed in the datacenter failure scenario is to point the DNS records to an active datacenter running a Waiter instance. As requests come into the available Waiter cluster, Waiter will spin up service instances on the new datacenter and process the requests.

Usage

To display usage instructions:

$ lein run --help

To sync tokens across multiple clusters, use comma separated names as follows:

$ lein run http://c1.localtest.me:9091 http://c2.localtest.me:9093
...
syncing tokens on clusters: \#{http://c1.localtest.me:9091 http://c2.localtest.me:9093}
...
completed syncing tokens
{:details ...
 :summary {:sync {:failed ...,
                  :unmodified ...,
                  :updated ...},
           :tokens {:num-processed ..., :total ...}}} }
exiting with code 0

Alternatively, it can also be run after building the uberjar (see below):

$ java -jar /path/to/token-syncer.jar http://c1.localtest.me:9091 http://c2.localtest.me:9093
...
syncing tokens on clusters: \#{http://c1.localtest.me:9091 http://c2.localtest.me:9093}
...
completed syncing tokens
{:details ...
 :summary {:sync {:failed ...,
                  :unmodified ...,
                  :updated ...},
           :tokens {:num-processed ..., :total ...}}} }
exiting with code 0

Implementation

The token syncer communicates with Waiter clusters on their respective /token endpoints to retrieve and manage individual tokens. Outline of steps performed by the syncer:

  1. Loads all tokens on each cluster.
  2. Filters out any tokens that have matching etags and are not deleted on all clusters (these tokens are already synced).
  3. Determines the latest version of the tokens using the token last updated times.
  4. Perform the sync operations using following cases:
    1. Report a conflict in sync if the token roots are different.
    2. Update all clusters with stale tokens to the latest token.
    3. Hard-deletes tokens on all clusters once all clusters agree that a token has been soft-deleted.

Build Uberjar

$ lein uberjar
...
Created /path-to-waiter-token-syncer/target/uberjar/token-syncer-0.1.0-SNAPSHOT.jar
Created /path-to-waiter-token-syncer/target/uberjar/token-syncer-0.1.0-SNAPSHOT-standalone.jar

Running Tests

Unit testing

$ lein test

Integration testing

Ensure you have multiple Waiters running, e.g. locally on port 9091 and 9093

$ WAITER_URIS="http://c1.localtest.me:9091,http://c2.localtest.me:9093" lein test :integration