Live Indexer

[Ullaakut]

Description

This epic is about creating a new binary in Flow DPS, which fulfills the same purpose as flow-dps-indexer, except that it reads from a live network instead of using historical data as a source.

Instead of running once and outputting a complete index, this binary is intended to keep running for as long as the network it's connected to is running, and to continuously fetch live updates.

Required Data

In order for the index to be complete, the Flow Live DPS Indexer will require:

• Trie Updates: Can be retrieved from an S3 Bucket that contains Ledger WAL segments
• Blocks: Can be retrieved using the consensus follower engine

[Ullaakut]

Following the meeting with the @dapperlabs team, as a first iteration what we will do is simply to fetch trie updates from an S3 bucket and feed them to the mapper. This feature will need the following set of tasks to be implemented:

• Make a component to read trie updates from an S3 bucket (See S3 Bucket Downloader POC #328)
• Make the Mapper FSM able to handle latency and to wait for new elements to be available (See [Live Indexer] Tolerate latency and network failures #336)
• Make the DPS able to follow the consensus of a Flow network in order to get block headers (See [Live Indexer] Receive blocks from the synchronization and consensus follower engines #337)
• Create a new binary for live indexing that takes the URL to an S3 bucket (See S3 Bucket Downloader POC #328)

Q/A:

• Should we mount the S3 bucket on the filesystem to use the current Feeder or is it better for us to implement a custom component for interacting with S3?
  • @awfm9: That approach strikes me as problematic, because it moves configuration to the system level.

[awfm9]

This has been done in #361 and #407.