Skip to content

Project Setup (Scala)

Jump to bottom
Alex Lopatin edited this page Jan 28, 2019 · 49 revisions

This page shows how to setup Flashbot in a brand new Scala project.

Looking for Java instructions? Go to Project Setup (Java)

Contents

  1. Create an SBT project
  2. Add the Flashbot dependency
  3. Configuration
  4. Launch a simple TradingEngine

The source code for this tutorial can be found at:

1. Create an SBT project

Start a new SBT project. An easy way to do this is with the scala-seed Giter8 template, like this:

# This tutorial uses the project name "Flashbot Starter Project", which initializes
# the project in a new directory named "flashbot-starter-project".
$ sbt new scala/scala-seed.g8

Ensure everything is working by navigating to the directory and running the project. This should print "hello" to the console.

$ cd flashbot-starter-project
$ sbt run
[info] ... a bunch of logging ...
[info] Running example.Hello
hello
[success] Total time: 1 s, completed Jan 16, 2019 12:59:58 PM

2. Add the Flashbot dependency

Edit your build.sbt file per the installation instructions in the README. This includes adding the Flashbot repository as well as the flashbot-client and flashbot-server dependencies. This also automatically pulls in Akka as a transitive dependency, which we'll use in our Main class.

Additionally:

  1. Add fork in run := true to your project settings. This allows sbt run to exit properly.
  2. Remove the organization and organizationName settings.

The final build.sbt should look like this.

3. Configuration

Flashbot (and Akka) use Typesafe Config for configuration. Create a file named application.conf in src/main/resources and add the following lines:

flashbot {
  engine-root = target/flashbot/engines
}

akka {
  loglevel = INFO
}

This configures Flashbot to use the <project_root>/target/flashbot/engines directory for all TradingEngine persistence. This is where all bot state is saved. Note that this is the data storage for the TradingEngine itself, which is separate from the SQL database (PostgresQL or H2) that we'll be using to store market data. Check out the reference config to see all the defaults.

4. Launch a simple TradingEngine

First, if you used the Giter8 template, delete the example package that comes with it:

$ rm -rf src/main/scala/example
$ rm -rf src/test/scala/example

Create a new Scala file (src/main/scala/MarketDataDashboard.scala) that will hold our main app. In it, we'll create a new actor system named "example-system" and a TradingEngine actor with the default props. Then we'll ping the engine to ensure that is started properly. Here's what MarketDataDashboard.scala should look like:

import akka.actor.ActorSystem
import com.infixtrading.flashbot.client.FlashbotClient
import com.infixtrading.flashbot.core.FlashbotConfig
import com.infixtrading.flashbot.engine.TradingEngine

import scala.concurrent.Await
import scala.concurrent.duration._

object Engine extends App {
  // Load config
  val config = FlashbotConfig.load

  // Create the actor system and trading engine
  val system = ActorSystem("example-system", config.conf)
  val engine = system.actorOf(TradingEngine.props("example-engine"))

  // Create a FlashbotClient
  val client = new FlashbotClient(engine)

  // Ping the trading engine.
  // All client methods have an async version that return a future instead of blocking.
  // In this case, that would be: client.pingAsync() => Future[Pong]
  val pong = client.ping()

  // Log engine start time to stdout
  println(s"Engine started at: ${pong.startedAt}")

  // Gracefully shutdown the system and exit the program.
  val term = Await.ready(system.terminate(), 5.seconds)
  System.exit(if (term.value.get.isSuccess) 0 else 1)
}

5. Check the engine for signs of life

Executing sbt run should result in the following output:

$ sbt run
[info] ... logging ...
[info] Engine started at: 2019-01-17T23:32:00.533Z
[success] Total time: 5 s, completed Jan 17, 2019 5:32:01 PM

6. Connect to market data from Coinbase

Now that setup is out of the way, we'll see first-hand how easy it is to build a real-time market data dashboard (including historical data!) by letting Flashbot ingest data directly from an exchange.

Our goal will be to acquire the following data sets for the both the BTC-USD and ETH-USD products on Coinbase:

  • Order books (real-time feed)
  • Trades (historical AND real-time feed)
  • 1-min price data (historical)

The main thing we need to do is add the flashbot.sources and flashbot.ingest properties to the configuration file (src/main/resources/application.conf). Here's what the additional configs should look like in order to ingest data described above:

flashbot {
  ...
  sources {
    # Declares the "btc_usd" and "eth_usd" pairs for the "coinbase" data source.
    coinbase.sources = ["btc_usd", "eth_usd"]
  }

  ingest {
    # Enables live-data ingest for the following data paths:
    #   - "coinbase/btc_usd/book"
    #   - "coinbase/btc_usd/trades"
    #   - "coinbase/eth_usd/book"
    #   - "coinbase/eth_usd/trades".
    enabled = ["coinbase/*/book", "coinbase/*/trades"]

    # Enables historical data backfills for the following data paths:
    #   - "coinbase/btc_usd/candles_1m"
    #   - "coinbase/btc_usd/trades"
    #   - "coinbase/eth_usd/candles_1m"
    #   - "coinbase/eth_usd/trades".
    backfill = ["coinbase/*/candles_1m", "coinbase/*/trades"]

    # Declares that we're not interested in 1-min price data, trade data, and order book data
    # that's older than 180 days, 90 days, and 7 days, respectively. Flashbot's data ingest
    # process will not request data outside of those time ranges, and will also actively delete
    # data that falls out of it's retention period.
    retention = [
      ["*/*/candles_1m", "180d"],
      ["*/*/trades", "90d"],
      ["*/*/book", "7d"]
    ]
  }
  ...
}

The above works because Coinbase is a built-in data source. If you implement your own data source, or use a 3rd party one, the same config pattern should work as long as the data source supports the markets and data types we specify. Flashbot will complain they don't.

The last step is to start a DataServer so that it can perform the actual data ingest based on the config we just created. It only takes one line of code! Go back to MarketDataDashboard.scala and add the following line right below val system = ...:

// Spin up the data server
val dataServer = system.actorOf(DataServer.props(config))

Now we have the necessary code that runs data ingest. But how do we inspect it? We'll start by requesting it through the FlashbotClient, which is hooked up to the TradingEngine. So to link everything together, let's connect the trading engine to the data server. Change the line where we create the trading engine from this:

val engine = system.actorOf(TradingEngine.props("example-engine", config))

to this:

val engine = system.actorOf(TradingEngine.props("example-engine", config, dataServer))

So far, we have declared what data we need in application.conf, created a DataServer that will get that data for us, connected it to our main TradinEngine, and connected to the TradingEngine with a FlashbotClient.

We will want to keep the program running so that the client can poll & print the market data as it's coming in from the data server. Right above the last three lines of MarketDataDashboard.scala where we exit the system, replace the client.pong() code with the following:

// Poll and print trades for 10 seconds.
val done = client.pollingMarketData[Trade]("coinbase/btc_usd/trades")
  .runForeach(println)
Await.ready(done, 10 seconds)

The source code of file in it's final state can be found here:

At last, execute sbt run from the command line again. This program will print every live "btc_usd" trade that for 10 seconds. The output of the program should look like this:

$ sbt run
...
... todo: output
...

7. Open the dashboard in Grafana

If you haven't yet, install Grafana. This guide assumes that it's running on localhost:3000 with the default login params (admin/admin). Otherwise you can update any of the following config keys:

flashbot {
  grafana {
    host = "localhost"
    port = 3000
    username = "admin"
    password = "admin"
  }
}

Go to http://localhost:3000 and select the "Market Data" dashboard from the left main panel.

8. Setup PostgreSQL database for persisting data (optional)

You may be wondering where the market data is being saved, since we haven't setup a database yet. Flashbot DataServers always save data to a SQL database and is configured by default to use an in-memory embedded H2 database. This means that all data will be lost once the program exits. While this is useful in development, we'll need to data to disk in most cases.

Flashbot includes two different database configurations: h2 and postgres. The default config is h2, but you can change this by setting the flashbot.db property in application.conf:

flashbot {
  db = "postgres"
}

By default Flashbot expects the PostgreSQL database to be on localhost:5432 with the database name "flashbot". Username and password are also "flashbot"/"flashbot". These settings are configurable with the dbHost, dbPort, dbName, dbUser, and dbPass properties. These defaults are all listed near the bottom of the reference.conf.

You can also create your own database configurations in your application.conf similar to the included ones. Then you can enable one of them by setting the flashbot.db setting to their property key.

9. Next steps

Now we have a market data dashboard that is created and managed by Flashbot. In the next section, we'll use other Grafana dashboard types to run backtests.

Clone this wiki locally