-
Notifications
You must be signed in to change notification settings - Fork 302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cometindex: 📇 add a minimal indexer prototype #4610
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cratelyn
added
C-enhancement
Category: an enhancement to the codebase
A-indexing
Area: Relates to event indexing.
labels
Jun 13, 2024
7 tasks
cratelyn
force-pushed
the
kate/cometindex
branch
from
June 13, 2024 20:01
f9a00ae
to
1c312db
Compare
cratelyn
force-pushed
the
kate/cometindex
branch
from
June 13, 2024 20:03
1c312db
to
13eedb7
Compare
opened #4611 to track the follow-on work related to |
used to confirm that polling works properly when running locally against a devnet.
cratelyn
force-pushed
the
kate/cometindex
branch
from
June 14, 2024 13:05
13eedb7
to
114afcd
Compare
5 tasks
@hdevalence this is ready for review, when you are able to take a look |
aubrika
approved these changes
Jun 24, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe your changes
this is a continuation of #4573.
Adds the skeleton of a minimal but extensible indexer. The indexer consumes events from a source (raw event) database and writes to a destination database with compiled / processed tables. The indexer is a library that other binaries import and slot their indexes into.
The
cometindex
library is intended to be generic over any cometbft events. Thepindexer
binary is for penumbra specifically and we can add all our own code to it. This structure mirrors the existing distinction between comet (generic) and the app. The code is intended to be shaped so that it's easy for users to plug in additional indexes. In particular, all of the "meat" of the binary is in thecometindex::Indexer
type, which builds an indexer and handles all arg parsing. This way, the external interface of the program, how it's invoked, etc, is the same no matter what app views are present, and orchestration code like containers doesn't need to be concerned with them.Before merging:
Index
trait toAppView
(it's a view of some application data)create_tables
method toinit_chain
pindexer
AppView
traits are public so thatpindexer
can be pulled in as a librarypindexer
crate that adds anIndexer::with_default_penumbra_app_views()
method that hooks in all theAppView
impls, so we can add to it as we fill in implsFollowup:
init_chain
to take aserde_json::Value
and add plumbing to theIndexer
builder to take agenesis.json
file in a standardized way so that every instance of the binary doesn't need to implement that -- ideally this is the exactgenesis.json
that was used by comet, and the plumbing in theIndexer
builder pulls out theapp_state
key and saves it to be able to pass it into eachAppView
. Longer term we could explore other options but for now I think it's simplest if we just make it a required parameter every time.Closes #4604
Checklist before requesting a review
If this code contains consensus-breaking changes, I have added the "consensus-breaking" label. Otherwise, I declare my belief that there are not consensus-breaking changes, for the following reason: