- Make sure the Java 8 JDK and
sbtare installed on your machine.
- Export the
JAVA_HOMEenvironment variable with the path to the JDK installation.
Configuring the Migrator
config.yaml for your migration using the template
config.yaml.example in the repository root. Read the comments throughout carefully.
Running on a live Spark cluster
The Scylla Migrator is built against Spark 2.4.4, so you'll need to run that version on your cluster.
build.sh, copy the jar from
./target/scala-2.11/scylla-migrator-assembly-0.0.1.jar and the
config.yaml you've created to the Spark master server.
Then, run this command on the Spark master server:
spark-submit --class com.scylladb.migrator.Migrator \ --master spark://<spark-master-hostname>:7077 \ --conf spark.scylla.config=<path to config.yaml> \ <path to scylla-migrator-assembly-0.0.1.jar>
Running the validator
This project also includes an entrypoint for comparing the source table and the target table. You can launch it as so (after performing the previous steps):
spark-submit --class com.scylladb.migrator.Validator \ --master spark://<spark-master-hostname>:7077 \ --conf spark.scylla.config=<path to config.yaml> \ <path to scylla-migrator-assembly-0.0.1.jar>
To run in the local Docker-based setup:
- First start the environment:
docker-compose up -d
cqlshin Cassandra's container and create a keyspace and a table with some data:
docker-compose exec cassandra cqlsh <create stuff>
cqlshin Scylla's container and create the destination keyspace and table with the same schema as the source table:
docker-compose exec scylla cqlsh <create stuff>
config.yamlfile; note the comments throughout.
spark-submitin the master's container to run the job:
docker-compose exec spark-master /spark/bin/spark-submit --class com.scylladb.migrator.Migrator \ --master spark://spark-master:7077 \ --conf spark.driver.host=spark-master \ --conf spark.scylla.config=/app/config.yaml \ /jars/scylla-migrator-assembly-0.0.1.jar
spark-master container mounts the
./target/scala-2.11 dir on
/jars and the repository root on
/app. To update the jar with new code, just run
build.sh and then run