Skip to content
Preview version of an open source tool that enables developers to run their AWS DynamoDB™ workloads on Apache Cassandra™. With the proxy, developers can run DynamoDB workloads outside of AWS (including on premises, other clouds, and in hybrid configurations).
Java Scala Other
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
common/cassandra/conf-dir/resources/cassandra/conf k8s k8s k8s Sep 12, 2019
conf oss driver Sep 11, 2019
docs gke Sep 13, 2019
gke gke Sep 13, 2019
k8s-local k8s k8s k8s Sep 12, 2019
migrate japanese docs Sep 12, 2019
src support for sort keys in query Sep 24, 2019
.gitignore gitignore Sep 12, 2019
.travis.yml travis docker Sep 18, 2019
Dockerfile Cleanup docker build Sep 12, 2019
LICENSE Fix headers. Add ASLv2 Sep 12, 2019
README-JP.md Update Japanese README Sep 12, 2019
README.md travis build Sep 19, 2019
docker-compose.yaml Cleanup docker build Sep 12, 2019
pom.xml support for sort keys in query Sep 24, 2019

README.md

dynamo-cassandra-proxy

Build state: build

dynamo-cassandra-proxy consists of a scalable proxy layer that sits between your app and Apache Cassandra. See the design summary in docs

It provides compatibility with the DynamoDB SDK which allows existing DynamoDB applications to read and write data to Cassandra without application changes.

It also supports the ability to sync DynamoDB tables with cassandra via DynamoDB Streams.

Config

Create your yaml based on the template:

cp conf/dynamo-cassandra-proxy.yaml.template conf/dynamo-cassandra-proxy.yaml

The following are the options supported by the proxy:

Option Description
streamsEnabled When set to true it enables the proxy to pull live data from an existing dynamodb table
dynamoRegion Only needed when streaming is enabled, region your dynamodb table is in
dyanmoAccessKey Only needed when streaming is enabled, used to connect to dynamodb streams
dyanmoSecretKey Only needed when streaming is enabled, used to connect to dynamodb streams
awsDynamodbEndpoint Only needed when streaming is enabled, used to connect to dynamodb streams
contactPoints Contact points to connect to Apache Cassandra(TM) cluster. If you are using the docker option just leave localhost
dockerCassandra When set to true it will stand up Cassandra in your local docker. Ensure the docker deamon is installed and running and your user has access to run docker ps

To run locally

Clone:

git clone git@github.com:datastax/dynamo-cassandra-proxy.git

Build:

mvn package

Run the app: whether you point the proxy at your own c* cluster or you rely on the proxy to stand up a cassandra node using the cassandraDocker option in the yaml. You can run the code locally by:

java -Ddw.contactPoints="$contactPoints" -cp target/dynamodb-cassandra-proxy-0.1.0.jar com.datastax.powertools.dcp.DCProxyApplication server conf/dynamo-cassandra-proxy.yaml

The proxy will come up and listen on port :8080. In your dynamodb application, just point your app to <host>:8080 in the SDK. A sample connection string (in Java) should look as follows:

        ClientConfiguration config = new ClientConfiguration();
        config.setMaxConnections(dynamodbMaxConnections);;
        String dynamodbEndpoint = "localhost:8080"
        String signinRegion = "dummy"
        AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder
                .EndpointConfiguration(protocol + "://" + dynamodbEndpoint, signinRegion);
        ddbBuilder = AmazonDynamoDBClientBuilder.standard()
                .withClientConfiguration(config)
                .withEndpointConfiguration(endpointConfiguration);

Note, MaxConnections is the main lever to get the AWS SDK to perform beyond very basic levels. We have tested this up to the maxium of 50 and it appears to scale almost linearly all the way up on a medum sized box. If you are doing some benchmarking and are looking to try to saturate a cassandra cluster, crank this value up.

To run via docker-compose

Build the app

mvn package

Build and run the docker containerst st

docker-compose up

To run in local k8s

Set up cassandra config map:

kubectl create configmap cassandra-config \

--from-file=common/cassandra/conf-dir/resources/cassandra/conf

Apply k8s yaml:

kubectl apply -f k8s-local/proxy-suite.yaml 

At this point, your pods should look as follows:

$ kubectl get pods                                                                                     [2:34:13]
NAME                  READY   STATUS              RESTARTS   AGE
cassandra-0           1/1     Running             0          2m35s
cassandra-1           1/1     Running             0          168s
cassandra-3           1/1     Running             0          123s
dynamo-cass-proxy-0   1/1     Running             4          63s

To terminate your deploymet run:

kubectl delete -f k8s-local/proxy-suite.yaml 

To run on GKE

Set up configMap

kubectl create configmap cassandra-config \
    --from-file=common/cassandra/conf-dir/resources/cassandra/conf

Apply k8s yaml:

kubectl apply -f gke/proxy-suite.yaml 

The workload will appear on your google console like so (ensure you're viewing the right project):

gke

To terminate your deploymet run:

kubectl delete -f gke/proxy-suite.yaml 

Contributing

A good place to start might be fleshing out your own Translator. For details on translators see Translators in the docs

MVP Roadmap:

Check means currently completed:

  • CreateTable - Done in json_blob
  • DeleteItem - Done in json_blob
  • DeleteTable - Done in json_blob
  • GetItem - Done in json_blob
  • PutItem - Done in json_blob
  • Query - Simple Case done in json_blob
  • Scan
  • Hybrid functionality - DDB to Cassandra
  • Hybrid functionality - Cassandra to DDB

Other features not yet implemented:

  • UpdateItem
  • BatchGetItem
  • BatchWriteItem
  • DescribeStream
  • DescribeTable
  • DescribeLimits
  • DescribeTimeToLive
  • GetRecords
  • GetShardIterator
  • ListStreams
  • ListTables
  • ListTagsOfResource
  • TagResource
  • UntagResource
  • UpdateTable
  • UpdateTimeToLive

License

This project is licensed under the Apache Public License 2.0

You can’t perform that action at this time.