Skip to content

ref(spans): Track cumulative latency in buffer logger#107317

Merged
fpacifici merged 7 commits intomasterfrom
fpacifici/record_outliers
Jan 30, 2026
Merged

ref(spans): Track cumulative latency in buffer logger#107317
fpacifici merged 7 commits intomasterfrom
fpacifici/record_outliers

Conversation

@fpacifici
Copy link
Contributor

@fpacifici fpacifici commented Jan 30, 2026

Changes BufferLogger to track cumulative latency instead of maximum latency, providing better visibility into which project/trace combinations accumulate the most latency over time.

I think tracking the cumulative time spent by outlier traces is going to be more valuable than specific traces that take longer in specific instances of the EVALSHA call.
This will allow us to keep constant track of outlier traces so we can take action wether manually (killswitch) or automatically (demote the trace to BLQ).

The idea is to keep track of the top traces and print the top 50 periodically.
Then identifying outliers will become easy.

Changes

  • Remove threshold filtering: All entries are now tracked regardless of latency value
  • Track cumulative latency: Store sum of all latencies for each project_and_trace instead of just the maximum value
  • Smart trimming: After each batch, if entries exceed 50, trim to keep only the top 50 by cumulative latency (removes lowest latency entries)
  • Sort by cumulative latency: Periodic logging (every 1 minute) now reports top 50 entries sorted by cumulative latency

Made with Cursor

fpacifici and others added 2 commits January 29, 2026 17:44
Changes BufferLogger to track cumulative latency instead of maximum latency:
- Remove threshold filtering - all entries are now tracked
- Track cumulative latency per project_and_trace instead of max latency
- Trim to top 1000 entries by cumulative latency after each batch
- Log top 50 entries by cumulative latency every minute
- Update tests to reflect new behavior

This provides better visibility into which project/trace combinations
accumulate the most latency over time rather than just peak values.

Co-authored-by: Cursor <cursoragent@cursor.com>
@fpacifici fpacifici requested review from a team as code owners January 30, 2026 01:54
@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Jan 30, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
Copy link
Contributor

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

@fpacifici fpacifici merged commit 700c3fa into master Jan 30, 2026
71 checks passed
@fpacifici fpacifici deleted the fpacifici/record_outliers branch January 30, 2026 22:58
priscilawebdev pushed a commit that referenced this pull request Feb 2, 2026
Changes `BufferLogger` to track cumulative latency instead of maximum
latency, providing better visibility into which project/trace
combinations accumulate the most latency over time.

I think tracking the cumulative time spent by outlier traces is going to
be more valuable than specific traces that take longer in specific
instances of the EVALSHA call.
This will allow us to keep constant track of outlier traces so we can
take action wether manually (killswitch) or automatically (demote the
trace to BLQ).

The idea is to keep track of the top traces and print the top 50
periodically.
 Then identifying outliers will become easy.

## Changes

- **Remove threshold filtering**: All entries are now tracked regardless
of latency value
- **Track cumulative latency**: Store sum of all latencies for each
project_and_trace instead of just the maximum value
- **Smart trimming**: After each batch, if entries exceed 50, trim to
keep only the top 50 by cumulative latency (removes lowest latency
entries)
- **Sort by cumulative latency**: Periodic logging (every 1 minute) now
reports top 50 entries sorted by cumulative latency

Made with [Cursor](https://cursor.com)

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions bot locked and limited conversation to collaborators Feb 15, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants