No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
scripts
snapshots
sql
README.md

README.md

EOS ZMQ plugin receiver tools

This is a set of scripts that are designed to receive and process the data flow that is generated by nodeos ZeroMQ plugin. The plugin exports every action on the blockchain, and also balances of assets and accounts involved in every transaction:

https://github.com/cc32d9/eos_zmq_plugin

The tools are written in Perl, and they require a MySQL server.

It's recommended to set up a separate server for MySQL data, isolated from the one running nodeos daemon, for better stability and storage management. As of end of July 2018, the whole MySQL database occupies over 100GB, and it's constantly growing. You need to set up a discarding procedure to delete old data in order to fit into your storage capacity.

As of end of July 2018, the full re-play of the EOS mainnet blockchain takes about 2-3 days, delayed mostly because of MySQL and disk I/O performance.

Installation

sudo apt-get install cpanminus gcc g++ libzmq5-dev mariadb-server \
libmysqlclient-dev libdbi-perl libexcel-writer-xlsx-perl \
libjson-xs-perl libjson-perl libencode-perl libcompress-lz4-perl

sudo cpanm DBD::MariaDB
sudo cpanm ZMQ::Raw


sudo mysql <sql/eos_zmq_dbcreate.sql

zmq_hub.pl

This script is a simple hub that connects to zmq_plugin socket on an nodeos server and copies the messages to a number of specified PUB or PUSH sockets. This allows publishing the data via PUB socket, while storing the stream of sata into an MySQL database.

perl scripts/zmq_hub.pl --pull=tcp://eos.example.com:5556 --push=tcp://127.0.0.1:6002 --pub=tcp://0.0.0.0:6003

eos_zmq_dumper.pl

This is a simple script that connects to a ZMQ PUB or PUSH socket and prints the received JSON data nicely formatted. You can use it for debugging or troubleshooting.

perl scripts/eos_zmq_dumper.pl  --sub=tcp://127.0.0.1:6003 | tee ~/eos.log

eos_zmq_dbwrite.pl

This script receives the EOS network events from a ZMQ PULL socket and stores the data into MySQL database.

The database scheme is available in sql/eos_zmq_dbcreate.sql.

perl scripts/eos_zmq_dbwrite.pl --connect=tcp://127.0.0.1:6002

eos_zmq_acc_report.pl

This script retrieves all actions and balance updates from the MySQL database for a specified account and produces an Excel spreadsheet file with a full report for specified period. By default, it makes a 3-month report for quarterly accounting.

perl scripts/eos_zmq_acc_report.pl --acc=cc32dninexxx --out=$HOME/2018Q2_cc32dninexxx.xlsx --start=2018Q2

Copyright and License

Copyright 2018 cc32d9@gmail.com

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Donations and paid service

ETH address: 0x7137bfe007B15F05d3BF7819d28419EAFCD6501E

EOS account: cc32dninexxx