Fetchers

To feed our market simulator, the system implements various fetchers. For live orders and transaction data, we have implemented fetchers for three Bitcoin exchange platforms. The Twitter fetcher from the Big Data course project has been adapted to this project’s framework and allows to receive Tweets in a push approach.

##Bitcoin Exchange Fetchers Fetchers for BTC-e, Bitstamp and Bitfinex exchange platforms have been extended and adapted from the previous project to this project’s framework. All of them have very similar APIs allowing to fetch latest transactions and current order books content using a pull approach. The data is in JSON format and is extracted using the liftweb library. In order to feed orders to the simulator’s exchange, the set of previous orders is maintained in the Bitcoin fetcher. When a new order books arrives, the difference between the current books content and the previous one is computed in order to filter the new and deleted orders. In order to implement a pull fetcher, the class must extend PullFetch[T] and override its interval() and fetch() methods which represent the frequency of the pull requests and the API access method.

trait Fetch[T]
abstract class PullFetch[T] extends Fetch[T] {
  def fetch(): List[T]
  def interval(): Int
}

The fetcher is then used as a PullFetchComponent constructor parameter.

implicit val builder = new ComponentBuilder("btceSystem")
val btceTransactionPullFetcher = new BtceTransactionPullFetcher
val btceTransactionFetcher =
  builder.createRef(Props(classOf[PullFetchComponent[Transaction]],
  btceTransactionPullFetcher, implicitly[ClassTag[Transaction]]), "btceTransactionFetcher")

##Twitter Fetcher The Twitter fetcher was imported from the previous project and adapted to the framework’s API. It uses the twitter4j library to fetch Tweets in a push approach. The filter is the following list of hashtags: ”bitcoin”, ”cryptocurrency”, ”btc”, ”bitcoins”. When receiving a tweet, it is classified by a manually trained Naive Bayes classifier.

The example in TwitterFlowTesterWithStorage shows how to setup a system to print live Tweets and store them in a Persistor.