EOS ZMQ plugin receiver tools
This is a set of scripts that are designed to receive and process the data flow that is generated by nodeos ZeroMQ plugin. The plugin exports every action on the blockchain, and also balances of assets and accounts involved in every transaction:
The tools are written in Perl, and they require a MySQL server.
It's recommended to set up a separate server for MySQL data, isolated from the one running nodeos daemon, for better stability and storage management. As of end of July 2018, the whole MySQL database occupies over 100GB, and it's constantly growing. You need to set up a discarding procedure to delete old data in order to fit into your storage capacity.
As of end of July 2018, the full re-play of the EOS mainnet blockchain takes about 2-3 days, delayed mostly because of MySQL and disk I/O performance.
sudo apt-get install cpanminus gcc g++ libzmq5-dev mariadb-server \ libmysqlclient-dev libdbi-perl libexcel-writer-xlsx-perl \ libjson-xs-perl libjson-perl libencode-perl libcompress-lz4-perl sudo cpanm DBD::MariaDB sudo cpanm ZMQ::Raw sudo mysql <sql/eos_zmq_dbcreate.sql
This script is a simple hub that connects to
zmq_plugin socket on an
nodeos server and copies the messages to a number of specified PUB or
PUSH sockets. This allows publishing the data via PUB socket, while
storing the stream of sata into an MySQL database.
perl scripts/zmq_hub.pl --pull=tcp://eos.example.com:5556 --push=tcp://127.0.0.1:6002 --pub=tcp://0.0.0.0:6003
This is a simple script that connects to a ZMQ PUB or PUSH socket and prints the received JSON data nicely formatted. You can use it for debugging or troubleshooting.
perl scripts/eos_zmq_dumper.pl --sub=tcp://127.0.0.1:6003 | tee ~/eos.log
This script receives the EOS network events from a ZMQ PULL socket and stores the data into MySQL database.
The database scheme is available in
perl scripts/eos_zmq_dbwrite.pl --connect=tcp://127.0.0.1:6002
This script retrieves all actions and balance updates from the MySQL database for a specified account and produces an Excel spreadsheet file with a full report for specified period. By default, it makes a 3-month report for quarterly accounting.
perl scripts/eos_zmq_acc_report.pl --acc=cc32dninexxx --out=$HOME/2018Q2_cc32dninexxx.xlsx --start=2018Q2
Copyright and License
Copyright 2018 email@example.com
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Donations and paid service