Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HDX-134] Metrics need to handle counter restarts and related edge cases better #120

Closed
MikeShi42 opened this issue Nov 21, 2023 · 1 comment · Fixed by #121
Closed

[HDX-134] Metrics need to handle counter restarts and related edge cases better #120

MikeShi42 opened this issue Nov 21, 2023 · 1 comment · Fixed by #121
Labels
bug Something isn't working Medium priority Created by Linear-GitHub Sync
Milestone
v.11

Comments

@MikeShi42
Copy link
Contributor

Right now we have a few conditions where we output NaN where we likely can do a bit better. Counter restarts is one edge case, and transitioning between two time series in the runningDifference calculation is another I believe.

Related code: https://github.com/hyperdxio/hyperdx/blob/main/packages/api/src/clickhouse/index.ts#L792-L805

From SyncLinear.com | HDX-134
@MikeShi42 MikeShi42 added bug Something isn't working Medium priority Created by Linear-GitHub Sync labels Nov 21, 2023
@MikeShi42
Copy link
Contributor Author

I think this gets exacerbated if a service is rapidly scaling up/down as it'll create transitions between two time series, and worse of all this gets worse on bigger granularities rather than smaller I think.

@jaggederest jaggederest mentioned this issue Nov 21, 2023
Merged
@MikeShi42 MikeShi42 modified the milestones: v.10, v.11 Nov 27, 2023
@kodiakhq kodiakhq bot closed this as completed in #121 Dec 5, 2023
kodiakhq bot pushed a commit that referenced this issue Dec 5, 2023
@jaggederest
Fixing returning NaN when rate metrics change / reset counter (#121)
141fce0 
fixes #120 

Intended to handle the case where a nonsensical rate would be returned (e.g. massively negative) or the metric `_string_attributes` change mid-query, to either clamp to zero or filter out possible NaN being currently returned. This should create cleaner graphs for rate metrics, allowing the rate to interpolate over missing data instead of creating a gap in the graph
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Medium priority Created by Linear-GitHub Sync
Projects
None yet
Milestone
v.11
Development

Successfully merging a pull request may close this issue.

Fixing returning NaN when rate metrics change / reset counter
1 participant
@MikeShi42