Tanglebeat is a configurable software agent with the primary purpose of collecting IOTA Tangle-related metrics to Prometheus Time Series Database. Historical metrics data later can be retrieved and displayed with such tools as Grafana or directly with Prometheus API.
Tanglebeat can be run in various standalone and distributed configurations to ensure high availability and objectivity of the metrics.
Tanglebeat is a result of experimenting with different approaches to how to measure a Tangle in objective and reliable way.
It was originally developed as standalone project. The site with live metrics can be found at tanglebeat.com
Currently Tanglebeat it is being developed under the umbrella of joined UNIO Project funded by Ecosystem Development Fund of IOTA Foundation.
- Functions What can it be useful for?
- Contents of the repository
- Download and install
- Configure and run
- Output stream
- Metrics exposed to Prometheus
A hub for IRI ZMQ streams
Tanglebeat collects messages from several IRI ZMQ inputs (configurable) and produces one output message stream. The resulting output represents operational state of the part of the network represented by those inputs. Inputs can cover some small cluster of the network or even one node, for example belonging to one operator (or one Economic Cluster in the future). It can also be very wide thus representing network as a whole. For example tanglebeat.com usually listens to up to 50 nodes.
Tanglebeat produces output from incoming data by using some kind of quorum based algorithm. In short: a message is passed to the output only after been received from several different ZMQ sources. Quorum parameter is usually 2 or 3 or more but can also be 1 (in case of single input ZMQ stream)
Output stream is used to calculate metrics. This way metrics reflects the performance and health of the network from the point of view of the cluster being listened to.
ZMQ sources can go down or up, nodes syncing or going out of sync, other nodes restarting and so on. By requiring evidences from several sources Tanglebeat filters out garbage messages which are not propagated by the network or the cluster of nodes.
Applications which rely on IRI ZMQ data may want to use Tanglebeat as ZMQ source independent from any specific IRI node.
Output message stream is exposed for external consumption by other programs (configurable). Find here how to use it.
A collector of ZMQ-based metrics
ZMQ-based metrics are usual metrics like TPS (transactions per second), CTPS (confirmed transactions per scond), Conf.rate (confirmation rate), duration between milestones. Tanglebeat also collects value based metrics such as number of confirmed transfers and value of confirmed transfers.
Tanglebeat exposes all metrics to the Prometheus instance of your choice or even several of them. Prometheus is times series database intended for collections of time series data (timed sequences of floating point numbers). It provides REST API and PromQL formula language to access stored time series data in every practically imaginable way.
Prometheus is often used as a data source for Grafana, a tool to display metrics in user friendly way: charts, gauges and similar.
A collector of non-ZMQ metrics
Some useful IOTA metrics can't be calculated from ZMQ data or it is more practical to collect it in another way. These metrics are submitted to Prometheus as well.
A separate module of Tanglebeat called TBSender does active sending of funds in IOTA ledger and calculates transfer statistics: transfers per hour, average transfer time, interval of expected transfer time and others.
It does it by sending and confirming/promoting few iotas worth transfers from one account to another in the endless loop, along addresses of the same seed. Several seeds (sequences) are used for that in parallel.
Transfers Per Hour or
TfPHmetrics is calculated the following way:
- (number if confirmed transfers completed in last hour in all sequences) / (average number of active sequences in the last hour)
Average number of active sequences is used to adjust for possible downtimes.
Average transfer time is calculated from transfer statistics.
Transfer confirmation time is estimated by taking 25 and 75 percentiles of real transfer confirmations times in the last hour.
Network latency. Promotion transactions are sent to the network to promote transfers. Tanglebeat records time when sent transaction returns from one of ZMQ streams back and after averaging collects it as a metrics.
A ZMQ state monitor
Tanglebeat exposes endpoint which returns states of all input ZMQ streams. Thus many nodes can be monitored at once: by up/down status, sync status, tps, ctps and conf. rate and other parameters.
The data on input streams is exposed using
Tanglebeat consists of two programs: tanglebeat itself and tbsender. The former can be run alone. tbsender is a separate (optional) program which does transfers to calculate non-ZMQ metrics. It sends all necessary data to tanglebeat instance which in turn submits metrics to Prometheus.
tanglebeatcontains Go package for the executable of main tanglebeat program.
tbsendercontains Go package for the executable of the tbsender.
examples/nano2zmqcontains code of
nano2zmqprogram, see Output stream how to use it.
examples/readnanocontains example how to read output of the Tanglebeat in the form of Nanomsg data stream.
libcontains shared packages. Some of them can be used as independent packages in other Go projects
lib/confirmercontains library for promotion, reattachment and confirmation of any bundle.
lib/multiapicontains library for IOTA API calls performed simultaneously to several nodes with automatic handling of responses. Redundant API calling is handy to ensure robustness of the daemon programs by using several IOTA nodes at once.
Download and install
Download and install Go
You will need Go compiler and environment to compile Tanglebeat binaries on your
platform. It is easier than it sounds.
nano2zmq) doesn't have binary dependencies on other packages, it is pure Go program.
To install Go follow the instructions.
Make sure to define
GOPATH environment variable to the root where all your
Go projects and/or dependencies will land.
GOPATH directory should contain at least
src (for sources) and
(for executable binaries) subdirectories.
PATH to your
To download main tanglebeat package with dependencies run:
go get -d github.com/unioproject/tanglebeat/tanglebeat/tanglebeat
To download tbsender package with dependencies run:
go get -d github.com/unioproject/tanglebeat/tbsender
To download nano2zmq package run:
go get -d github.com/unioproject/tanglebeat/examples/nano2zmq
To download readnano package run:
go get -d github.com/unioproject/tanglebeat/examples/readnano
Compile Tanglebeat binaries
GOPATH/src/github.com/unioproject/tanglebeat/tanglebeat current and run
GOPATH/src/github.com/unioproject/tanglebeat/tbsender current and run
GOPATH/src/github.com/unioproject/tanglebeat/examples/nano2zmq current and run
go install (C dependencies!)
GOPATH/src/github.com/unioproject/tanglebeat/examples/readnano current and run
The above will produce respective executable binaries in
Configure and run
Configure and run Tanglebeat instance
It is run with the command:
tanglebeat [-cfg <config file>]
Tanglebeat instance is configured via YAML config file. It is expecting
in the current directory or in the directory specified by
It also can be specified with
-cfg <config file> command line flag.
examples/config contains example of the config file.
Please read instructions right in the file. In most cases you'll only need to adjust ports used
by the instance and static list of URI's of IRI ZMQs you want your instance to listen to.
Another important part is
quorumToPass parameter. It controls how many evidences from different sources is needed
to pass the message to the output. If you run tanglebeat with one input, it must be 1. Otherwise it must be 2 or more.
For example if you set it to
5 each fifth message with the same hash will be pushed to the output
while messages which, for some reason, are circulating among 4 nodes only will be filtered out.
Note, that Prometheus is needed for Tanglebeat only if you want to store metrics. It is not needed if you use it only as a message hub.
Any Prometheus instance can collect (scrape) metrics from Tanglebeat instance. You need to install one only if you don't have one yet. Otherwise you need:
- to specify
scrape targetin the config file of the Prometheus instance.
- to provide
tanglebeat.rulesto the Prometheus (only if you run TBSender)
Follow instruction to download and install Prometheus.
Metrics of the Tanglebeat will be exposed to Prometheus with endpoint
/metrics on the port, configured in
scrape target must be specified in the configuration file of the Prometheus.
Here we also provide tanglebeat.rules
file which has to be specified for the Prometheus instance to calculate
tfph_adjusted metrics if you want to use TBSender
for TfPH metrics. See Prometheus docs.
If you want to display Prometheus metrics, you can install Grafana server by following these instructions.
TBSender is run only if you need TfPH and network latency metrics. It is run by command
TBSender is expecting configuration file
tbsender.yml in the current directory or in the directory
SITE_DATA_DIR environment variable.
Please find here an example of tbsender.yml
It contains detailed comments of how to set up configuration parameters. In general, you'll need to specify following things to set up the environment of TBSender:
Global parameters for sequences: URLs of IRI nodes, API timeouts and similar.
Specify sequences. Eeach sequence of addresses is defined by
index0parameters. Sequence of addresses is security level 2 addresses with the
seedstarting at specified index. TBSender will be sending iotas along those sequences: whole balance of the current address to the next address and so on in a loop.
Each sequence can override global parameters if needed: nodes, promo strategy, timeouts etc.
Output can be enabled/disabled the following way:
iriMsgStream: outputEnabled: true outputPort: 5500
The following types of IRI ZMQ messages are available from the output Nanomsg stream:
lmi(latest milestone changed)
lmhs(latest solid milestone hash).
We are using Nanomsg as output for technical reasons (which may become irrelevant in the future). Meanwhile, if you want to stick to ZMQ as as transport, we provide Nanomsg to ZMQ converter.
After compiled (see above), it can be run by the command:
nano2zmq [-from <input nanoimsg URI> [ -to <output ZMQ port> ] ]
By default it reads Nanomsg input
tcp://tanglebeat.com:5550 and sends it to the port
as ZMQ stream.
nanomsg -from tcp://localhost:5550 will read Nanomsg on 5550 and produce ZMQ output on 5556.
nano2zmq uses this Zero MQ library for Go and so it has
binary dependencies with original ZeroMQ version 4.0.1 which must be
installed (Tanglebeat itself don't have this dependency).
Metrics exposed to Prometheus
The following metrics are exposed to Prometheus by Tanglebeat. It can be found in Grafana admin frontend when designing charts and panels.
tx(transaction) messages. Used to calculate TPS metrics
sn(confirmation) messages. Used to calculate CTPS metrics and confirmation rate.
tanglebeat_transfer_volume_counter_prod- counter of transferred amount in iotas.
It sums up approximately calculated volume of transfer. It is calculated by summing up values of positive transactions in the bundle less transactions which do not move iotas in the bundle, i.e. those address which net sum is 0 (fake transfers) and less assumed transfer to reminder address. The reminder is assumed equal to value of the last transaction in the bundle if it is positive and it does not belong to the fake transfer. Otherwise it is 0.
tanglebeat_transfer_counter_prodcounter of confimed bundles with positive moved volume of iotas.
tanglebeat_miota_price_usdIOTA price as taken form Coincap site
tanglebeat_echo_firsttime in miliseconds when first echo of the transaction, send by TBSender, is seen from ZMQ inout.
tanglebeat_echo_lasttime in seconds when last echo of the transaction, send by TBSender, comes form all ZMQ input.
tanglebeat_lm_conf_rate_30minconfirmation rate as provided by Luca Moser. Is is based on statistics collected while sending zero value transactions and obeserving it's confirmation.
Transfers Per Hourmetrics. Average number of confirmed transfer one sequence of TBSender was able to make in last 1 hour