New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Local blocks processor query all blocks #2480
Local blocks processor query all blocks #2480
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM
if p.headBlock == nil { | ||
return traceql.FetchSpansResponse{}, fmt.Errorf("head block is nil") | ||
// Blocks to check | ||
blocks := make([]common.BackendBlock, 0, 1+len(p.walBlocks)+len(p.completeBlocks)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is going to create a lot of memory pressure with each read by copying memory around. We could loop through the blocks directly instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. I wonder if we could keep this blocks
object on the struct, since we're already locking, we could keep it updated on the iteration but not need to re-allocate new memory.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I took this approach because it greatly simplifies the below logic and error handling, one loop instead of two or three (head block, wal blocks, complete blocks). The overhead of this slice should be minimal - expect a few dozen blocks and single digit requests per second to this api.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me.
What this PR does:
This PR is part of a wider initiative to perform on-demand metrics calculations. Updates the
/generator/api/metrics
API to query all blocks. Includes some basic caching with simplelru that definitely needs to be replaced with something more robust. Finally added a test but it's a good one 😬Which issue(s) this PR fixes:
Fixes n/a
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]