Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a LedgerBackend for benchmarking ingestion performance as a function of ledger size #5480

Open
tamirms opened this issue Sep 30, 2024 · 0 comments
Assignees

Comments

@tamirms
Copy link
Contributor

tamirms commented Sep 30, 2024

Given the following configuration options:

  • minimum number of transactions / operations per ledger
  • ledger close time
  • queue capacity

The LedgerBackend will start prefetching ledgers from the data lake and, using #5479 , it will combine consecutive ledgers until we have a ledger which meets the minimum number of transactions / operations per ledger. Then the resulting "fat" ledger will be emitted at a rate matching the configured ledger close time (we can default to 5 seconds per ledger). If the fat ledgers are emitted at a rate faster than they can be consumed, the LedgerBackend will push the fat ledgers onto a queue. Once the queue has exceeded its capacity we can terminate with an error because we know that the consumer of the LedgerBackend is not able to keep up with the current throughput settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants