Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor trade aggregations #5169

Open
sreuland opened this issue Jan 18, 2024 · 0 comments
Open

Refactor trade aggregations #5169

sreuland opened this issue Jan 18, 2024 · 0 comments
Labels
feature request objective-8 performance issues aimed at improving performance

Comments

@sreuland
Copy link
Contributor

What problem does your feature solve?

trade aggregations is implemented within horizon, it is intertwined into ingestion, db, and api. horizon ingestion must spend extra time doing trade aggregation tasks after each live ledger is ingested and during reingestion ranges to keep the trade agg tables fresh, this adds overhead to ingestion processing times and also introduces complexities for parallel worker ingestion such as #5127.

What would you like to see?

refactor trade aggregations for horizon benefits, there are two options from least to most effort:

  • change the trade aggregation bucket to align per ledger rather than a unit of time that could span ledgers as that prevents parallel worker reingestion on contiguous ranges - see comment here
  • pull out trade aggregation from horizon and package as a standalone microservice, I think this becomes a viable option to consider when pre-compiled remote tx-meta, #4911 becomes available, the trade agg microservice can follow stream topology pattern of running as a processor using the tx-meta as input stream and derive the trade agg data models from there, no traditional ingestion needed.

What alternatives are there?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request objective-8 performance issues aimed at improving performance
Projects
Status: Backlog
Development

No branches or pull requests

2 participants