Solana Indexer: Process messages in parallel#292
Merged
rickyrombo merged 6 commits intomainfrom Aug 4, 2025
Merged
Conversation
Contributor
rickyrombo
commented
Aug 2, 2025
rickyrombo
commented
- Adds parallelism to solana-indexer.
- Defaults to 50 workers for stage and prod and 1 for local
- Tested with 50 workers locally and indexed USDC, wAUDIO, BONK, WEN and was able to catch up from behind 3000 slots. Was seeing numbers of around 50k account balance changes indexed per minute with this many workers. We'll see how our cloud sql db handles this - can always tune the worker count - but this should give us plenty of headroom I think.
- Reduces slot checkpoint writes to every 10 slots and only if the slot is greater than the current one to help reduce load.
- Bails early in processing if found signature in the cache, as it's already processed then.
- Removes sql transaction that wasn't doing anything since it wasn't wired up properly. (Fixing the code to use it causes deadlocks as the concurrency causes the inserts to compete for locks on tables I guess?)
schottra
approved these changes
Aug 4, 2025
| AudiusdURL string | ||
| ChainId string | ||
| BirdeyeToken string | ||
| SolanaIndexerWorkers int |
Contributor
There was a problem hiding this comment.
lol okay maybe my only gripe with the golang aligned formatting thing 😆
| MaxSupportedTransactionVersion: &rpc.MaxSupportedTransactionVersion0, | ||
| }, | ||
| ) | ||
| }, 5, 1*time.Second) |
Contributor
There was a problem hiding this comment.
nit: worth using a named constant somewhere
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.