Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement streaming_service.py #1

Closed
medvedev1088 opened this issue Sep 17, 2018 · 0 comments
Closed

Implement streaming_service.py #1

medvedev1088 opened this issue Sep 17, 2018 · 0 comments

Comments

@medvedev1088
Copy link
Member

medvedev1088 commented Sep 17, 2018

Streaming service

  1. Continuously polls for new blocks using Ethereum JSON RPC API (e.g. every 10 seconds).
  2. Outputs data to GooglePubSubItemExporter, which publishes data to PubSub.
  3. Saves the last synchronized block to last_sync_block.txt file on every period.

This script can be used as a basis https://github.com/blockchain-etl/ethereum-etl/blob/feature/streaming/stream.py

================================

PubSub publisher guide https://cloud.google.com/pubsub/docs/publisher

There is some code in this fork https://github.com/airswap/ethereum-etl/commit/cc7f86e139d317452c9cf0c975b99feb6ed8d7a0#diff-a8e6a25bc32464a59378bac1befac9eb

TODO: Consider using geth subscriptions to handle chain reorganizations https://github.com/ethereum/go-ethereum/wiki/RPC-PUB-SUB#newheads

@medvedev1088 medvedev1088 changed the title Add streaming_service.py Implement streaming_service.py Sep 17, 2018
@medvedev1088 medvedev1088 moved this from To do to Done in Ethereum ETL Streaming Jul 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

No branches or pull requests

1 participant