Implements a stats tracker for various stats derived from the Bitcoin
Stats are stored in a Postgres database and as JSON files.
Additionally, tracks mempool stats using
The dashboard is publicly available at dashboard.bitcoinops.org.
The program logs all statistics from getblockstats, and some statistics derived from these (e.g. percentage of transactions that have quality X).
The exact statistics tracked are those set in the struct definition for
An example JSON file with an explanation of the statistics stored is STATS_TRACKED.md
expand-getblockstats branch of https://github.com/bitcoinops/bitcoin with extended getblockstats RPC.
dashboard-rpc branch of https://github.com/bitcoinops/btcd for RPC client that can use the extended getblockstats RPC.
go-pg as a Postgres client.
dashboard-rpc branch of btcd before running
Set environment variables for Postgres
DB the name of the database,
DB_USERNAME Postgres username,
DB_PASSWORD Postgres password,
DB_ADDR if using a remote database. If not specified
DB_ADDR defaults to "http://localhost:5432" which is the default address for a local Postgres instance.
You don't have to specify a table, or create one yourself. This set of programs uses
go-pg, which we use to create tables with schemas derived from Go struct definitions.
Set environment variables for bitcoind RPC access
which should correspond to
rpcpassword in the bitcoin.conf file.
BITCOIND_HOST which defaults to "localhost:8332"
Quick environment variable setup
Copy the file
example_env_file.txt and edit to match your local configuration.
export $(cat example_env_file.txt |xargs -L 1) should set environment variables to match the file.
Assumes previously mentioned environment variables are set, and correctly correspond to running instances of
./btc-dashboard [OPTIONAL: -recovery] [OPTIONAL: -start=X] [OPTIONAL: -end=Y] [-workers=N]
Modes of Operation
-mempoolSetting this flag starts a mempool tracker that continuously stores data derived from RPCs into a database. It does not halt by itself, but is safe to stop (catches SIGINT and SIGTERM after all writes are finished).
-recoveryStarts workers on any progress files left over from previously unfinished runs.
-insert-jsonUploads contents of every JSON file in the default directory and uploads them into Postgres.
-json=[true,false]If set, every
DashboardDatastruct inserted into the database will also be saved as a JSON file. Defaults to
true. The default directory is
log.Fatalis called). Requires
EMAIL_PASSWORDto be set for sending email account, and
RECIPIENT_EMAILS(comma-separated list of email addresses) for all recipients.
-workers=N flag will cause the program to start
N different RPC clients to do its work. The default value is 2.
-end=Y flags are used to specify the range of blockheights to analyze: [X, Y).
Not using the
-end=Y flag will cause the program to do a live analysis. In this case, if a starting height is specified the live analysis will start at that height. Otherwise it starts 6 blocks behind the current blockheight of the chaintip. The analysis stays 6 blocks behind the tip in order to avoid implementing re-org logic.
The live analysis processes blocks one at a time with a single-worker as they come in.
Otherwise with at least the
-end flag set, the program starts a backfill analysis from the interval [start, end), where start defaults to 0.
Tracking Progress and Recovering from Failures
Because back-filling a database with the statistics from the entire Bitcoin blockchain can take a while, this program also implements some basic features to track progress of workers and features to recover from program failures.
Whenever a worker thread starts processing a range of blocks, it will write its progress to a file in a
worker-progress directory. The file states the starting blockheight, the last blockheight analyzed, and the ending blockheight for this specific worker.
Suppose you ran the command
./btc-dashboard -start=1000 -end=2000 -workers=2
and stopped the program before it completed. In the
worker-progress directory you might see two files that have names similar to:
with contents that look something like:
Start=1000 Last=1234 End=1500
If you would like to restart the program continuing where these last workers left off, you can just run the command:
./btc-dashboard -recovery -workers=2
which will start 2 workers to finish up this work, which will continue to mark their progress in the same files.
Workers that complete their assigned work delete their progress files.
Results from the database can be plugged into Grafana for visualization.