Skip to content
Mining cryptocurrencies on Apache Yarn with Apache Spark
Scala Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
src/main first commit Jan 1, 2018
.gitigonre first commit Jan 1, 2018
.travis.yml first commit Jan 1, 2018 Update Jan 1, 2018

Dogecoin meets Apache Spark !

CPU Mining cryptocurrencies using Apache Spark without any external dependencies.


Using CPU Miner from in an Apache Spark job that starts tasks on a Yarn/Mesos cluster. Each cluster executor starts an instance of CPU Miner using the provided mining options.


Mining pools

Get yourself an account in any mining pool that supports CPU mining :

Currency Mining pools

Keep an eye on and look for new coins that do not require special mining equipments.


Package the project into a single jar (requires SBT), in the project root run :

sbt package

You'll get a spark-yarn-miner_2.11-*.jar jar.


Next, use spark-submit to submit your job to the cluster :

spark-submit --master yarn --class fr.marouni.spark.coins.SparkYarnMiner spark-yarn-miner_2.11-0.1.0-SNAPSHOT.jar --url XXX --username YYY --worker-id WWW --password UUUU

Where :

  • --url : Your mining pool URL (check mining pool docs). Example : stratum+tcp://
  • --username : Mining pool account username
  • --worker-id : Mining pool worker id (check mining pool docs)
  • --password : Mining pool account password


Tune your mining with the following spark-submit options :

  • spark-submit --conf spark.executor.instances=10 on a 10 nodes cluster
  • spark-submit --conf spark.executor.cores=6 on a cluster with 6 cores per node

Recommended tuning : 1 executor per cluster node with maximum allowed number of cores per executor.

You can’t perform that action at this time.