Market Data Dashboard

5. Check the engine for signs of life

Executing sbt run should result in the following output:

$ sbt run
[info] ... logging ...
[info] Engine started at: 2019-01-17T23:32:00.533Z
[success] Total time: 5 s, completed Jan 17, 2019 5:32:01 PM

Now that setup is out of the way, we'll see first-hand how easy it is to build a real-time market data dashboard (including historical data!) by letting Flashbot ingest data directly from an exchange.

Our goal will be to acquire the following data sets for the both the BTC-USD and ETH-USD products on Coinbase:

• Order books (real-time feed)
• Trades (historical AND real-time feed)
• 1-min price data (historical)

The main thing we need to do is add the flashbot.sources and flashbot.ingest properties to the configuration file (src/main/resources/application.conf). Here's what the additional configs should look like in order to ingest data described above:

flashbot {
  ...
  sources {
    # Declares the "btc_usd" and "eth_usd" pairs for the "coinbase" data source.
    coinbase.sources = ["btc_usd", "eth_usd"]
  }

  ingest {
    # Enables live-data ingest for the following data paths:
    #   - "coinbase/btc_usd/book"
    #   - "coinbase/btc_usd/trades"
    #   - "coinbase/eth_usd/book"
    #   - "coinbase/eth_usd/trades".
    enabled = ["coinbase/*/book", "coinbase/*/trades"]

    # Enables historical data backfills for the following data paths:
    #   - "coinbase/btc_usd/candles_1m"
    #   - "coinbase/btc_usd/trades"
    #   - "coinbase/eth_usd/candles_1m"
    #   - "coinbase/eth_usd/trades".
    backfill = ["coinbase/*/candles_1m", "coinbase/*/trades"]

    # Declares that we're not interested in 1-min price data, trade data, and order book data
    # that's older than 180 days, 90 days, and 7 days, respectively. Flashbot's data ingest
    # process will not request data outside of those time ranges, and will also actively delete
    # data that falls out of it's retention period.
    retention = [
      ["*/*/candles_1m", "180d"],
      ["*/*/trades", "90d"],
      ["*/*/book", "7d"]
    ]
  }
  ...
}

The above works because Coinbase is a built-in data source. If you implement your own data source, or use a 3rd party one, the same config pattern should work as long as the data source supports the markets and data types we specify. Flashbot will complain they don't.

The last step is to start a DataServer so that it can perform the actual data ingest based on the config we just created. It only takes one line of code! Go back to MarketDataDashboard.scala and add the following line right below val system = ...:

// Spin up the data server
val dataServer = system.actorOf(DataServer.props(config))

Now we have the necessary code that runs data ingest. But how do we inspect it? We'll start by requesting it through the FlashbotClient, which is hooked up to the TradingEngine. So to link everything together, let's connect the trading engine to the data server. Change the line where we create the trading engine from this:

val engine = system.actorOf(TradingEngine.props("example-engine", config))

to this:

val engine = system.actorOf(TradingEngine.props("example-engine", config, dataServer))

So far, we have declared what data we need in application.conf, created a DataServer that will get that data for us, connected it to our main TradinEngine, and connected to the TradingEngine with a FlashbotClient.

We will want to keep the program running so that the client can poll & print the market data as it's coming in from the data server. Right above the last three lines of MarketDataDashboard.scala where we exit the system, replace the client.pong() code with the following:

// Poll and print trades for 10 seconds.
val done = client.pollingMarketData[Trade]("coinbase/btc_usd/trades")
  .runForeach(println)
Await.ready(done, 10 seconds)

The source code of file in it's final state can be found here:

At last, execute sbt run from the command line again. This program will print every live "btc_usd" trade that for 10 seconds. The output of the program should look like this:

$ sbt run
...
... todo: output
...

7. Open the dashboard in Grafana

If you haven't yet, install Grafana. This guide assumes that it's running on localhost:3000 with the default login params (admin/admin). Otherwise you can update any of the following config keys:

flashbot {
  grafana {
    host = "localhost"
    port = 3000
    username = "admin"
    password = "admin"
  }
}

Go to http://localhost:3000 and select the "Market Data" dashboard from the left main panel.

8. Setup PostgreSQL database for persisting data (optional)

You may be wondering where the market data is being saved, since we haven't setup a database yet. Flashbot DataServers always save data to a SQL database and is configured by default to use an in-memory embedded H2 database. This means that all data will be lost once the program exits. While this is useful in development, we'll need to data to disk in most cases.

Flashbot includes two different database configurations: h2 and postgres. The default config is h2, but you can change this by setting the flashbot.db property in application.conf:

flashbot {
  db = "postgres"
}

By default Flashbot expects the PostgreSQL database to be on localhost:5432 with the database name "flashbot". Username and password are also "flashbot"/"flashbot". These settings are configurable with the dbHost, dbPort, dbName, dbUser, and dbPass properties. These defaults are all listed near the bottom of the reference.conf.

You can also create your own database configurations in your application.conf similar to the included ones. Then you can enable one of them by setting the flashbot.db setting to their property key.

9. Next steps

Now we have a market data dashboard that is created and managed by Flashbot. In the next section, we'll use other Grafana dashboard types to run backtests.