initial release

.gitignore

@@ -0,0 +1,5 @@
+dbs
+node_modules
+*.swp
+*.log
+.DS_Store

README.md

@@ -1 +1,178 @@
-# bitfinex-terminal
+# Bitfinex Terminal
+
+Bitfinex Terminal offers market data as live data streams and is built on top of [Dazaar](https://github.com/bitfinexcom/dazaar). It offers first class support for Algo Traders by offering market data over a distributed database to everyone.
+
+The market data streams are free and allow algo traders and data scientist easy and fast access to our historical trading data. Trading data is easy to download and replicate. It is shared on a P2P network - like BitTorrent - but with a database interface. Also, unlike with BitTorrent, you won't have to download everything until you can use it, streaming time ranges is supported. And if you decide to download more data, e.g. from an earlier timeframe, it will of course preserve the data ordered, in a nice B-Tree, for fast access.
+
+## Table of Contents
+
+  1. [How to use it?](#howtouse)
+  1. [Support for Algo traders](#firstclass)
+  1. [Example: Get a full snapshot of the BTCUSD trades, and keep following new updates](#example-trades)
+  1. [Example: Query candles data](#example-candles)
+  1. [Tutorial: Backtest your Trading Strategies with Bitfinex Terminal & Honey Framework](./articles/backtesting-with-hf.md)
+  1. [Article: Learn more about Dazaar](https://blog.dazaar.com/2020/09/12/introducing-dazaar/)
+
+<a id="howtouse" />
+
+## How to use it?
+
+Every data stream in Bitfinex Terminal and also Dazaar has a unique `id` on the network. Imagine it like the url of the datastream. You can directly use ids or you can use dazaar cards, which are [available for download](../cards). We have prepared a few examples that will show how to use it.
+
+<a id="firstclass" />
+
+## First class support for Algo traders
+
+Bitfinex Terminal is also supported by the [Honey Framework](https://honey.bitfinex.com/). You can easily backtest your Honey Framework Strategies on Bitfinex Terminal with the [Honey Framework backtesting tools](./articles/backtesting-with-hf.md). And you can also decide to sell live trading signals over Dazaar with [bfx-hf-strategy-dazaar](https://github.com/bitfinexcom/bfx-hf-strategy-dazaar).
+
+## Examples
+
+<a id="example-trades" />
+
+Also make sure to check our [tutorial material](./articles) :)
+
+### Example: Get a full snapshot of the BTCUSD trades, and keep following new updates
+
+In our first example we will prepare a full copy of the Bitfinex Trades data. We also want to keep following new updates that come in. We will walk through a small set of code, as if we were writing it. The full example can be found at [examples/trades-full-copy.js](examples/trades-full-copy.js).
+
+As a first step we have to require our dependencies:
+
+```js
+const dazaar = require('dazaar')
+const swarm = require('dazaar/swarm')
+
+const Hyperbee = require('hyperbee')
+const keyEncoding = require('bitfinex-terminal-key-encoding')
+const terms = require('bitfinex-terminal-terms-of-use')
+```
+
+And we create a dazaar market. Our data will be stored at `dbs/full-trades`
+
+```js
+const market = dazaar('dbs/full-trades')
+```
+
+Then we download the [dazaar card for the stream](../cards/bitfinex.terminal.BTCUSD.trades.json) and load it. The option `live: true` keeps the connection open, so we keep listening for updates after the data is fully synced. If we would set `sparse` to `true`, we would just download requested data that we request in a query. This way, we make a full copy. We also accept the Bitfinex Terminal terms of use by loading them into dazaar:
+
+```js
+const card = require('../cards/bitfinex.terminal.BTCUSD.trades.json')
+const buyer = dmarket.buy(card, { live: true, sparse: false, terms })
+```
+
+In the next step a lot happens. We register an event listener for the `feed` event. When it is emitted, we know that the feed is ready for consuming. We also set up a `Hyperbee` instance, which will provide us a nice interface to access the data. We pass the `db` instance to a function called `doQuery` which we will highlight next.
+
+```js
+buyer.on('feed', function () {
+  console.log('got feed')
+
+  const db = new Hyperbee(buyer.feed, {
+    keyEncoding,
+    valueEncoding: 'json'
+  })
+
+  doQuery(db)
+})
+```
+
+The function `doQuery` will make a request for us. It will request the required data prioritized and print it to our console. We select all trades with a timestamp larger than October 2018, 9:00 UTC and `2019`. We also limit the results to `10`:
+
+```js
+function doQuery (db) {
+  db.createReadStream({
+    gte: { timestamp: new Date('2018-10-10T09:00:00.000Z') },
+    lte: { timestamp: new Date('2019-10-10T09:00:00.000Z') },
+    limit: 10
+  }).on('data', (d) => {
+    console.log(d)
+  })
+}
+```
+
+
+To start everything, we have to join the P2P swarm:
+
+```js
+swarm(buyer)
+```
+
+And for demonstration purposes we also log the downloaded elements, so we can see the progress:
+
+```js
+setInterval(() => {
+  if (!buyer.feed) return
+
+  console.log('data feed length is', buyer.feed.length, 'elements')
+}, 1000)
+```
+
+This example code will download the whole BTCUSD trades dataset from Bitfinex Terminal. If the connection is reset, it will just resume where it stopped. We also make a request for a timerange and print it to the console. The timerange is prioritized and downloaded first. When all downloading is finished, we keep the socket open to receive the latest updates. That is a lot thats done for us behind the scenes.
+
+<a id="example-candles" />
+
+### Example: Query candles data
+
+Querying Candle data is as easy as getting historical trade data. All candle data is consolidated into one database, and we just have to select our candle type.
+
+The full example can be found at [examples/candles-sparse-select.js](examples/candles-sparse-select.js). Our setup is similar to the trades example, we require our dependencies and set up a database that is stored on disk:
+
+```js
+const dazaar = require('dazaar')
+const swarm = require('dazaar/swarm')
+
+const Hyperbee = require('hyperbee')
+const keyEncoding = require('bitfinex-terminal-key-encoding')
+
+const market = dazaar('dbs/full-candles')
+```
+
+This time we load a Dazaar Card for candles. We also enable the `sparse` mode, that means, just the data we directly request is downloaded. We also accept the Bitfinex Terminal terms of use by loading module for it into Dazaar:
+
+```js
+const card = require('../cards/bitfinex.terminal.BTCUSD.candles.json')
+const terms = require('bitfinex-terminal-terms-of-use')
+const buyer = dmarket.buy(card, { sparse: true, terms })
+```
+
+Our event listener looks exactly the same as the one from the previous example:
+
+```js
+buyer.on('feed', function () {
+  console.log('got feed')
+
+  const db = new Hyperbee(buyer.feed, {
+    keyEncoding,
+    valueEncoding: 'json'
+  })
+
+  doQuery(db)
+})
+```
+
+For our query we define a timeframe and the candle type. We also reverse the results, so we should get the newest entries first:
+
+```js
+function doQuery (db) {
+  db.createReadStream({
+    gte: { candle: '5m', timestamp: new Date('2018-10-10T09:00:00.000Z') },
+    lte: { candle: '5m', timestamp: new Date('2019-10-10T09:00:00.000Z') },
+    limit: 10,
+    reverse: true
+  }).on('data', (d) => {
+    console.log(d)
+  })
+}
+```
+
+## Tutorials
+
+<a id="tutorials" />
+
+Our Tutorials can be found in the [./articles](./articles) folder.
+
+ - [Tutorial: Execute your Trading Strategy with the Honey Framework and Bitfinex Terminal](./articles/execute-strategy-hf.md)
+
+## Articles
+
+<a id="articles" />
+
+ - [Introducing Dazaar](https://blog.dazaar.com/2020/09/12/introducing-dazaar/)

articles/backtesting-with-hf.md

@@ -0,0 +1,199 @@
+# Backtest your Trading Strategies with Bitfinex Terminal & Honey Framework
+
+The Honey Framework provides easy ways to create and backtest trading strategies. Today we are taking a look at how you can backtest your trading strategies with Bitfinex Terminal data and Node.js. Bitfinex Terminal was released with the needs of algo traders in mind - a fast, reliable way to sync and share historical data. We will take a trading strategy, EMA Crossover, and backtest it on historical candle data.
+
+We will run our backtest on 5-minute candles from BTCUSD trading data. That data is, like in a blockchain, stored in a Merkle Tree and can be cryptographically verified. The data is shared over a peer-to-peer (P2P) network and can be live-streamed. That means the backtest can start to run while we are fetching the data and resume the downloads after a connection reset is done for us. All these features make Terminal a compelling choice for sharing trading data and trading signals.
+
+Bitfinex Terminal supports Dazaar Cards, which is an easy way to access the data streams. Every data stream has a unique ID, which is verified cryptographically, similar to a Bitcoin address. The Dazaar card contains a description of the content and the stream ID. Imagine it like a torrent file, but for encrypted, real-time data streams.
+
+
+## Let's try it
+
+
+To begin, we install the required dependencies:
+
+```
+npm install dazaar hyperbee bitfinex-terminal-key-encoding bfx-hf-util bfx-hf-backtest bfx-hf-strategy bitfinex-terminal-terms-of-use
+
+```
+
+We also have to create a file; let's say `backtest.js`. We can now start to write our code.
+
+To see the results of our backtest on the console we have to enable debug output:
+
+```js
+process.env.DEBUG = process.env.DEBUG || 'bfx:*'
+```
+
+The EMA-Cross-strategy we run today is one of the example strategies included in the Honey Framework. We load that strategy with our other dependencies. One dependency of it are the Terms of Use for Bitfinex Terminal - we will have to load them into Dazaar after reading and acknowledging them:
+
+
+```js
+const dazaar = require('dazaar')
+const swarm = require('dazaar/swarm')
+const Hyperbee = require('hyperbee')
+const keyEncoding = require('bitfinex-terminal-key-encoding')
+
+const HFBT = require('bfx-hf-backtest')
+const { SYMBOLS, TIME_FRAMES } = require('bfx-hf-util')
+const EMAStrategy = require('bfx-hf-strategy/examples/ema_cross')
+
+const terms = require('bitfinex-terminal-terms-of-use')
+```
+
+We also define a small helper function to give us the time from exactly 24 hours in the past:
+
+```js
+const get24HoursAgo = (date) => {
+  const res = date.getTime() - (1 * 86400 * 1000)
+
+  return new Date(res)
+}
+```
+
+For backtesting, we have to define the market we test and pass it to the EMA Strategy:
+
+```js
+const market = {
+  symbol: SYMBOLS.BTC_USD,
+  tf: TIME_FRAMES.ONE_HOUR
+}
+
+const strat = EMAStrategy(market)
+```
+
+Then we can initialise Dazaar:
+
+```js
+const dmarket = dazaar('dbs/terminal-backtest')
+```
+
+With the above command, Dazaar will create a local database in `dbs/terminal-backtest`. All of our data will be stored in this folder, so if you want to start over, you can simply delete it.
+
+As the next step we download the Dazaar Card for BTCUSD from [https://raw.githubusercontent.com/bitfinexcom/bitfinex-terminal/master/cards/bitfinex.terminal.btcusd.trades.json](https://raw.githubusercontent.com/bitfinexcom/bitfinex-terminal/master/cards/bitfinex.terminal.btcusd.trades.json) and load it:
+
+```
+wget https://raw.githubusercontent.com/bitfinexcom/bitfinex-terminal/master/cards/bitfinex.terminal.btcusd.trades.json
+```
+
+```js
+const card = require('./bitfinex.terminal.btcusd.candles.json')
+```
+
+Dazaar also supports paid feeds, e.g. for selling trading signals of a successful strategy, but the Bitfinex Terminal data is free. Our card is loaded into Dazaar. We enable sparse mode, with sparse mode set, it will just download data requested from us. We also load the terms of service we required above, after we read and acknowledged them:
+
+```js
+const buyer = dmarket.buy(card, { sparse: true, terms })
+```
+
+If we wanted to download a full copy of all candles in the background, we would set `sparse` to `false`.
+
+Once the data is ready, Dazaar will emit a `feed` event. After that is emitted, we can set up Hyperbee on top of the Dazaar feed. Hyperbee provides us with a B-Tree structure for easy querying of the data. For Terminal, we use a special key encoding module, which makes querying the data easier. We already loaded the key encoding module with `const keyEncoding = require('bitfinex-terminal-key-encoding')`. Once the Hyperbee database is set up, we can call `runTest` and work with the database:
+
+```js
+buyer.on('feed', function () {
+  console.log('got feed')
+
+  const db = new Hyperbee(buyer.feed, {
+    keyEncoding,
+    valueEncoding: 'json'
+  })
+
+  runTest(db)
+})
+```
+
+The last part of our backtest is the actual testing. For that, we have to define the function `runTest`:
+
+```js
+async function runTest (db) {
+
+}
+```
+
+Inside the function `runTest` we have to define the timeframe we want to use for our test. For the beginning of the timeframe, we use the helper function we created at the beginning of our tutorial. The value for `to` points to the current time:
+
+```js
+ const from = get24HoursAgo(new Date())
+ const to = new Date()
+```
+
+Together with our strategy and the market, we are passing the timeframe definitions to `HFBT.execStream`. It returns a function called `exec` and a function called `onEnd`:
+
+```js
+  const { exec, onEnd } = await HFBT.execStream(strat, market, {
+    from,
+    to
+  })
+```
+
+The function `exec` will be applied to each element of the stream, but first we have to initiate the stream. We will run our test on `5m` candles. To get all data within that 24 hour timeframe we use our variables `from` and `to` in the respective fields `lte` (less than equal) and `gte` (greater than equal):
+
+```js
+  const stream = db.createReadStream({
+    gte: { candle: '5m', timestamp: from },
+    lte: { candle: '5m', timestamp: to }
+  })
+```
+
+With the help of an Async Iterator we call exec on every entry and store the result in `btState`:
+
+```js
+  let btState
+  for await (const data of stream) {
+    const { key, value } = data
+    btState = await exec(key, value)
+  }
+```
+
+Then, once the iterator gives us no results any more, we call the function `onEnd`, which will print our results to the console:
+
+```js
+  await onEnd(btState)
+```
+
+Here is the whole function `runTest`:
+
+```js
+async function runTest (db) {
+  const from = get24HoursAgo(new Date())
+  const to = new Date()
+
+  const { exec, onEnd } = await HFBT.execStream(strat, market, {
+    from,
+    to
+  })
+
+  const stream = db.createReadStream({
+    gte: { candle: '5m', timestamp: from },
+    lte: { candle: '5m', timestamp: to }
+  })
+
+  let btState
+  for await (const data of stream) {
+    const { key, value } = data
+    btState = await exec(key, value)
+  }
+
+  await onEnd(btState)
+}
+```
+
+
+After finishing the function `runTest`, we are almost done. To start everything, we have to go online:
+
+```js
+swarm(buyer)
+```
+
+[Here is the whole code in one file for our backtest](../examples/backtest.js).
+
+Now, when we run our file, we get the result printed to the console:
+
+![results](../img/results.png)
+
+As you can see, we would have made a loss with this strategy during the timeframe 9/6/2020, 3:18:58 PM to 9/7/2020, 3:18:58 PM, when this article was written.
+
+## Conclusion
+
+In this article, we've taken a look at how backtesting a Honey Framework trading strategy works with Bitfinex Terminal. We were able to create a backtest in under 70 lines of code with an easy and reliable way to sync large amounts of data. Everything was running on top of Dazaar, which can be used to sell trading signals too. One of our next articles will cover exactly that topic, so stay tuned!

cards/bitfinex.terminal.btcusd.candles.json

@@ -0,0 +1,5 @@
+{
+  "name": "Bitfinex Terminal BTCUSD Candles Dazaar Card",
+  "id": "76ef1d766eee63ff473e33e1f231c3b65cf1042b1a353c027b48bdbeba00b969",
+  "payment": []
+}

cards/bitfinex.terminal.btcusd.trades.json

@@ -0,0 +1,5 @@
+{
+  "name": "Bitfinex Terminal BTCUSD Trades Dazaar Card",
+  "id": "03a3af100709be37800a189e64d51fe63cd62d6660ce9bf3e6fea3d6670c1310",
+  "payment": []
+}