This is a set of scripts that are designed to receive and process the data flow that is generated by nodeos ZeroMQ plugin. The plugin exports every action on the blockchain, and also balances of assets and accounts involved in every transaction:
https://github.com/cc32d9/eos_zmq_plugin
The tools are written in Perl, and they require a MySQL server.
It's recommended to set up a separate server for MySQL data, isolated from the one running nodeos daemon, for better stability and storage management. As of end of July 2018, the whole MySQL database occupies over 100GB, and it's constantly growing. You need to set up a discarding procedure to delete old data in order to fit into your storage capacity.
As of end of July 2018, the full re-play of the EOS mainnet blockchain takes about 2-3 days, delayed mostly because of MySQL and disk I/O performance.
sudo apt-get install cpanminus gcc g++ libzmq5-dev mariadb-server \
libmysqlclient-dev libdbi-perl libexcel-writer-xlsx-perl \
libjson-xs-perl libjson-perl libencode-perl libcompress-lz4-perl
sudo cpanm DBD::MariaDB
sudo cpanm ZMQ::Raw
sudo mysql <sql/eos_zmq_dbcreate.sql
This script is a simple hub that connects to zmq_plugin socket on an
nodeos server and copies the messages to a number of specified PUB or
PUSH sockets. This allows publishing the data via PUB socket, while
storing the stream of sata into an MySQL database.
perl scripts/zmq_hub.pl --pull=tcp://eos.example.com:5556 --push=tcp://127.0.0.1:6002 --pub=tcp://0.0.0.0:6003
This is a simple script that connects to a ZMQ PUB or PUSH socket and prints the received JSON data nicely formatted. You can use it for debugging or troubleshooting.
perl scripts/eos_zmq_dumper.pl --sub=tcp://127.0.0.1:6003 | tee ~/eos.log
This script receives the EOS network events from a ZMQ PULL socket and stores the data into MySQL database.
The database scheme is available in sql/eos_zmq_dbcreate.sql.
perl scripts/eos_zmq_dbwrite.pl --connect=tcp://127.0.0.1:6002
This script retrieves all actions and balance updates from the MySQL database for a specified account and produces an Excel spreadsheet file with a full report for specified period. By default, it makes a 3-month report for quarterly accounting.
perl scripts/eos_zmq_acc_report.pl --acc=cc32dninexxx --out=$HOME/2018Q2_cc32dninexxx.xlsx --start=2018Q2
Copyright 2018 cc32d9@gmail.com
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
ETH address: 0x7137bfe007B15F05d3BF7819d28419EAFCD6501E
EOS account: cc32dninexxx