Skip to content

Using Spark Scala Graphx to analyze the Bitcoin transaction graph

Jörn Franke edited this page Mar 3, 2018 · 17 revisions
Clone this wiki locally

This is a Spark/Graphx application written in Scala demonstrating some of the capabilities of the hadoopcryptoledger library. It takes as input a set of files on HDFS containing Bitcoin Blockchain data. It generates out of the data a Bitcoin transaction graph, i.e. a graph consisting of the vertices describing Bitcoin addresses and the edges represent transactions between these addresses.

It returns as a result from the graph the top 5 Bitcoin addresses in terms of number of transfers to them. Most of the times these addresses belong to so called Mixing services that try to obfuscate transactions in the Bitcoin blockchain.

It has successfully been tested with the Cloudera Quickstart VM 5.5 and HDP Sandbox 2.5, but other Hadoop distributions should work equally well. Spark 1.5 was used for testing.

Getting blockchain data

See here how to fetch Bitcoin blockchain data.

After it has been copied you are ready to use the example.

Building the example


git clone hadoopcryptoledger

You can build the application by changing to the directory hadoopcryptoledger/examples/scala-spark-graphx-bitcointransaction and using the following command:

sbt clean assembly test it:test

This will also execute the integration tests

You will find the jar "example-hcl-spark-scala-graphx-bitcointransaction.jar" in ./target/scala-2.10

Running the example

Make sure that the output directory is clean:

hadoop fs -rm -R /user/bitcoin/output

Execute the following command (to execute it using a local master)

spark-submit --class org.zuinnote.spark.bitcoin.example.SparkScalaBitcoinTransactionGraph --master local[8] ./target/scala-2.10/example-hcl-spark-scala-graphx-bitcointransaction.jar /user/bitcoin/input /user/bitcoin/output

After the Spark job has completed, you find the result in /user/spark/bitcoin/output. You can display it using the following command:

hadoop fs -cat /user/bitcoin/output/part-00000

An example for such an output can be found here (vertexId,(receiving Bitcoin address, number of inputs)):


Hint: You can put the addresses (without the prefix bitcoinaddress_) in popular Bitcoin explorers (e.g. to get more details about the address.

Of course, you can integrate other data together with the blockchain data or add to the edges in the Graph the amount for each transaction.

More Information


Understanding the structure of Bitcoin data:

Understanding the destination of Bitcoin addresses and why they cannot be compared with transactions occurring in the current financial system:



Heuristics for joining Bitcoin Transactions: Meiklejohn, Sarah; Pomarole, Marjori; McKoy, Damon; Jordan, Grant; Voelker, Geoffrey: "A Fistful of Bitcoins: Characterizing Payments Among Men with No Names" (Extended Abstract), Proceedings of the 2013 conference on Internet Measurement, Barcelona, Spain, 2013