BINGE (Binance INGEstor)

BINGE is a simple pipeline to ingest raw trades from Binance, into Clickhouse, using Vector to handle the insertion, and Grafana to power some simple visualizations.

Binge is a simplified version of internal tooling, which has been processing Binance trades, on and off, for several years. Binge is setup to subscribe to and ingest coins that are actively traded against USDT (BTCUSDT, ETHUSDT, etc..), however one can easily extend the coins ingested with minor tweaks to binge.py.

Setup

Docker

1. Bring up binge using docker compose:

   docker compose up

2. Connect to Grafana: http://localhost:3001

Manual Setup

Pre-requisites

• Clickhouse
• Grafana
  • Clickhouse-Datasource
• Python3
  • Poetry
• Vector

Steps

1. Import the the required tables and views into clickhouse:

   clickhouse-client --queries-file ./sql/clickhouse.sql

2. Configure vector with the appropriate endpoint and credentials information to sink into clickhouse.

3. Setup python and run binge:

   cd binge
   poetry install
   poetry shell
   python binge.py | vector --config-toml ../vector/vector.toml

4. Follow the official Import a Dashboard steps to import the example dashboards located in the grafana directory to your Grafana instance.

   Note

   To enable sub 5 second refresh rates, you will need to modify your grafana.ini min_refresh_interval setting to allow faster refresh rates.