Need for timestamp, because some sources may be with a latency

[sbadan]

Hello,
I'm using Telegraf, Prometheus and Grafana integration. And we get some data from cloudwatch into telegraf.
The problem is that on cloudwatch the data appears with a ~10 minutes delay. But telegraf sends the data in the current moment, and it's written in the Prometheus this way. So we have 10 minutes old data displayed as current.
You can read here about the cloudwatch delay: https://help.datadoghq.com/hc/en-us/articles/203826415-Are-my-AWS-CloudWatch-metrics-delayed-

Would it be possible to implement data timestamp into Prometheus, so that we would somehow send the timestamp along with the data? Or maybe implement some time manipulations in the queries, so that we could retrieve the data series with the desired delay?

[brian-brazil]

Prometheus itself already supports this, however client libraries (and thus the cloudwatch exporter) do not pending #398

[sbadan]

Hello,
Maybe I can make my own cloudwatch executable. The thing is that the telegraf cloudwatch plugin returns correct timestamps as from Cloudwatch:

[root@prod-monitoring telegraf]# telegraf -config ./telegraf.conf -input-filter cloudwatch -test

• Plugin: cloudwatch, Collection 1
• Internal: 5m0s
  Dimension [VolumeId]:[vol-b9e70c6e]

cloudwatch_aws_ebs,host=prod-monitoring,region=us-east-1,unit=count,volume_id=vol-b9e70c6e volume_read_ops_average=0,volume_read_ops_maximum=0,volume_read_ops_minimum=0,volume_read_ops_sample_count=1,volume_read_ops_sum=0 1483095000000000000
cloudwatch_aws_ebs,host=prod-monitoring,region=us-east-1,unit=count,volume_id=vol-b9e70c6e volume_write_ops_average=730,volume_write_ops_maximum=730,volume_write_ops_minimum=730,volume_write_ops_sample_count=1,volume_write_ops_sum=730 1483095000000000000

[root@prod-monitoring telegraf]# date +%s
1483095658

The difference between the current date (1483095658) and the date reported by telegraf cloudwatch plugin (1483095000000000000) is roughly 10 minutes. So on the plugin side everything looks correct.
But grafana displays the values with the current timestamp, not with the timestamp of 10 minutes ago.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.