-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
collector-local timestamps for gnmi plugin #8411
Comments
I think we're fine with this change as long as it's a configuration option that needs to be enabled to take effect. If you'd like to open a PR we'd be happy to take a look at it |
Sometimes it can be useful to use a local, on receive timestamp for gnmi notifications. This is the case for sample subscriptions. The gnmi spec requires that the update timestamps are the time when the value from the underlying data source (e.g. counter hardware) changed. This value might not change for a long time, resulting in a timestamp way back in the past. For such situations, it can be useful to use the current time on the server, turning the semantics of the timestamp into "This was the value on the device at time T according to the collector" from "This was when the value last changed on the device." Fixes influxdata#8411
Sometimes it can be useful to use a local, on receive timestamp for gnmi notifications. This is the case for sample subscriptions. The gnmi spec requires that the update timestamps are the time when the value from the underlying data source (e.g. counter hardware) changed. This value might not change for a long time, resulting in a timestamp way back in the past. For such situations, it can be useful to use the current time on the server, turning the semantics of the timestamp into "This was the value on the device at time T according to the collector" from "This was when the value last changed on the device." Fixes influxdata#8411
Sometimes it can be useful to use a local, on receive timestamp for gnmi notifications. This is the case for sample subscriptions. The gnmi spec requires that the update timestamps are the time when the value from the underlying data source (e.g. counter hardware) changed. This value might not change for a long time, resulting in a timestamp way back in the past. For such situations, it can be useful to use the current time on the server, turning the semantics of the timestamp into "This was the value on the device at time T according to the collector" from "This was when the value last changed on the device." Fixes influxdata#8411
Sometimes it can be useful to use a local, on receive timestamp for gnmi notifications. This is the case for sample subscriptions. The gnmi spec requires that the update timestamps are the time when the value from the underlying data source (e.g. counter hardware) changed. This value might not change for a long time, resulting in a timestamp way back in the past. For such situations, it can be useful to use the current time on the server, turning the semantics of the timestamp into "This was the value on the device at time T according to the collector" from "This was when the value last changed on the device." Fixes influxdata#8411
Sometimes it can be useful to use a local, on receive timestamp for gnmi notifications. This is the case for sample subscriptions. The gnmi spec requires that the update timestamps are the time when the value from the underlying data source (e.g. counter hardware) changed. This value might not change for a long time, resulting in a timestamp way back in the past. For such situations, it can be useful to use the current time on the server, turning the semantics of the timestamp into "This was the value on the device at time T according to the collector" from "This was when the value last changed on the device." Fixes influxdata#8411
Was this implemented? Is there documentation? Sorry if this is a dumb question but I'm a Github noob networking engineer. |
@flyerhawk there is this pending pull request #8420 I just tested it using Dockerfile:
We also need to configure this feature (disabled by default) in the conf file |
Hello! We found that having your timestamps set locally would be very useful across a lot of Telegraf plugins, not just gnmi. Instead of building this enhancement for gnmi specifically we created an issue to create a global tag setting to set your local timestamp #8689. We'd welcome any PRs for #8689. If you are looking to use this immediately for gnmi, please use the list of builds from @rski's PR #8420 (comment). Otherwise the global tag feature will be on our roadmap to add soon. |
Sometimes it can be useful to use a local, on receive timestamp for gnmi notifications. This is the case for sample subscriptions. The gnmi spec requires that the update timestamps are the time when the value from the underlying data source (e.g. counter hardware) changed. This value might not change for a long time, resulting in a timestamp way back in the past. For such situations, it can be useful to use the current time on the server, turning the semantics of the timestamp into "This was the value on the device at time T according to the collector" from "This was when the value last changed on the device." Fixes influxdata#8411
The solution for this will be to utilize the Starlark processor. There is this example script that can be added to use the current/local timestamp over the field device's timestamp. |
wow that's pretty awesome |
Feature Request
Proposal:
Timestamp gnmi updates using the collector's local time instead of using the timestamp field of the notification.
Current behavior:
Updates received over gnmi are always timestamped using the Timestamp field which comes fro mthe device.
Desired behavior:
Make it possible to use the collector's time when the update was received to timestamp updates.
Use case:
Depending on the device that sends gnmi updates, the timestamp semantics can be different. According to the spec, the process that streams gnmi notifications should use the time the modification happened, not the time it was seen by the process. For example, when the hardware counters change, the time when those counters changed is to be used.
This differs between implementations. Some devices stream a fresh timestamp every time they send an update to a client that is subscribed to a counter path, some use a fixed timestamp of the time the value came to be.
This results in two kinds of collected datum; One is a time series, where each timestamp from the update means counter was X at time T, with multiple values of T even if X did not change. In this case, you can find the current value by looking at the last update for the path which is at most the sample frequency. Looking at a window of T to T+(sample frequency) shows all the information about a device.
In the other, the timestamps could be at any point in the past, since they are the time at which a value was established. Looking at a window of T to T+(sample frequence) shows only the values that changed within that window.
Certain orgs prefer the first, spec breaking behaviour. This can be implemented on the collector in a device agnostic way by ignoring the timestamp provided by the device and recording the time an update was received.
I already have the code for this change, it's pretty simple to make it possible and configurable, but I thought I'd open an issue first to gauge the interest on this.
The text was updated successfully, but these errors were encountered: