-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Linkerd2 stat CLI and linkerd2 web dashboard delayed compared to Grafana #1453
Comments
@willtrking This is a great find! I agree that these should be consistent. I included a bunch of info below to help explain the discrepancy. We should use this issue to track standardizing so that the dashboard and the CLI present the same view of the data. When you run the
On the grafana dashboard, we instead use a query that looks like this:
These two queries are different in a couple of ways:
The main reason for these discrepancies is that the grafana dashboard shows you timeseries data that includes a value for every datapoint in a given time window. The stat command, on the other hand, shows you a single, rolled-up value for all data points in a time window. The default time window for the stat command is 1 minute, so it takes a full minute for changes in request volume to be fully reflected in the stat output. You can try shortening the time window for the stat command to only include the last two datapoints, and that should mirror the behavior of the grafana dashboard much more closely. You can do this via the
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions. |
@siggy @grampelberg do you think it's worth turning Kevin's excellent response above into documentation? |
@wmorgan i think so. the whole area under https://linkerd.io/2/cli/ needs love, this would be a good addition to that. |
Hey there,
When monitoring service RPS with
linkerd stat
and the linkerd dashboard, theRPS
statistic is super delayed comparatively to Grafana.E.g. if we have a spike of traffic from ~10 rps to ~100 rps for 10 seconds, Grafana properly responds and shows the spike in RPS.
linkerd stat
and the dashboard seem to respond here as well. However, once the RPS drops back down to ~10 after that 10 second period,linkerd stat
and the dashboard are delayed, taking sometimes over 30 seconds to adjust back down to what Grafana shows (which is accurate).The text was updated successfully, but these errors were encountered: