BSC ETL lets you convert blockchain data into convenient formats like CSVs and relational databases. Based around and uses the (ethereum-etl) as a library.
Full documentation available here.
Install BSC ETL:
pip3 install bsc-etl
Export blocks and transactions (Schema, Reference):
> bscetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://bsc-dataseed1.binance.org/v3/7aef3f0cd1f64408b163814b22cc643c
Export BEP20 transfers (Schema, Reference):
> bscetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/bsc/geth.ipc --output token_transfers.csv
Export traces (Schema, Reference):
> bscetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/bsc/parity.ipc --output traces.csv
Stream blocks, transactions, logs, token_transfers continually to console (Reference):
> pip3 install bsc-etl[streaming]
> bscetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://bsc-dataseed1.binance.org/v3/7aef3f0cd1f64408b163814b22cc643c
Find other commands here.
For the latest version, check out the repo and call
> pip3 install -e .
> python3 bscetl.py
- Schema
- Command Reference
- Documentation
- Public Datasets in BigQuery
- Exporting the Blockchain
- Querying in Amazon Athena
- Querying in Google BigQuery
- Querying in Kaggle
- Airflow DAGs
- Postgres ETL
> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> export PROVIDER_URL=<your_porvider_uri>
> pytest -vv
> pip3 install tox
> tox
-
Install Docker: https://docs.docker.com/install/
-
Build a docker image
> docker build -t bsc-etl:latest . > docker image ls
-
Run a container out of the image
> docker run -v $HOME/output:/bscetl-etl/output bscetl-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://bsc-dataseed1.binance.org > docker run -v $HOME/output:/bscetl-etl/output bscetl-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://bsc-dataseed1.binance.org
-
Run streaming to console or Pub/Sub
> docker build -t bsc-etl:latest -f Dockerfile . > echo "Stream to console" > docker run bscetl-etl:latest stream --start-block 500000 --log-file log.txt > echo "Stream to Pub/Sub" > docker run -v /path_to_credentials_file/:/bsc-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/bsc-etl/credentials_file.json bscetl-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_bsc
** Add Some! **