Skip to content
JavaFX and commandline application to import events from the Ethereum blockchain into ElasticSearch, MongoDB, Hazelcast, CQEngine and SQLite.
Java CSS
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
gradle/wrapper
src/main
.gitignore
.travis.yml
LICENSE
README.md
application.yaml
build.gradle
eth-ingest.jpg
gradlew
gradlew.bat

README.md

ethereum-ingest Build Status

Imports blocks and transactions from the Ethereum blockchain into ElasticSearch, MongoDB, Hazelcast, CQEngine and SQLite! Works with Geth IPC, Geth REST API and Infura.

Demo video: YouTube

user interface

Tested with

  • Java 11 (minimim requirement)
  • ElasticSeach 7.0.0
  • MongoDB 4.0.8
  • Hazelcast 3.10.5
  • geth 1.8.23

Building

Build with

# detects the current platform.
./gradlew build

# to build for a specific platform, 'win', 'mac' or 'linux'.
./gradlew build -Pplatform=linux

Requires chili-core through jitpack or local repo.

Importing

The first step is to start your ethereum IPC client, for geth use:

geth --rpcapi personal,db,eth,net,web3 --rpc --testnet

Start the importer with:

java -jar <filename>.jar --import
java -jar <filename>.jar --gui
java -jar <filename>.jar --help
  • --import: starts an import using application.json.
  • --gui: starts the application with the graphical user interface.

Configuring

Set configuration in application.json before running --import. WHen using the graphical application the configuration is saved automatically.

Default configuration

{
  "startBlock" : "1964770",
  "blockEnd" : "1964900",
  "storage" : "ELASTICSEARCH",
  "targetNode" : "\\\\.\\pipe\\geth.targetNode",
  "os" : "WINDOWS",
  "txIndex" : "ether-tx-demo-iix",
  "blockIndex" : "etdder-block-demo-iix",
  "backpressureBlocks" : 6,
  "backPressureTx" : 32,
  "txImport" : true,
  "blockImport" : true
}

Backpressure for blocks and tx should not exceed 200 when multiplied with eachother. If you want to increase these values further you need to make sure the storage is capable of handling that many connections.

Storage can be any of the following

  • MONGODB (default: localhost:27017)
  • ELASTICSEARCH (default: localhost:9300)
  • HAZELCAST
  • SQLITE (CQEngine)
  • MEMORY

os can be any of the following, required for targetNode to work correctly

  • UNIX
  • WINDOWS

Imports can be executed multiple times over the same block range without resulting in duplicates.

To configure a custom host:port for MongoDB or ElasticSearch please add/edit this file in "conf/system/storage.yaml"

---
storage:
  com.codingchili.core.storage.MongoDBMap:
    host: "localhost"
    port: 27017
  com.codingchili.core.storage.ElasticMap:
    host: "localhost"
    port: 9200

Contributing

Submit an issue or a PR ! 💙

donate

You can’t perform that action at this time.