rate behaves differently than in Prometheus

[kalgary]

Describe the bug
Hi,
I'm evaluating the use of Victoria metrics as long term storage, my goal is to get rid of Prom federation.

I have noticed a different behavior when the same query is run against prom and victoria metrics.

rate(process_cpu_seconds_total{job="$instance-kafka1"}[1m])

VICTORIA METRICS

PROMETHEUS

stored time series seems to contain the same data:

process_cpu_seconds_total{job="$instance-kafka1"}

VICTORIA METRICS

PROMETHEUS

Configuration on Prometheus side:

remote_write:
- url: http://10.8.24.62:8428/api/v1/write
  remote_timeout: 30s
  queue_config:
    capacity: 100000
    max_shards: 30
    max_samples_per_send: 10000
    batch_send_deadline: 5s
    max_retries: 10
    min_backoff: 30ms
    max_backoff: 100ms

Expected behavior
I expect to see the same graph when querying the two different backends from grafana, Prom or Victoria metrics should produce the same trends, but as you can see in the graphs, this is not the case.

Version

$ ./victoria-metrics-prod --version
victoria-metrics-20190813-235905-tags-v1.25.0-0-g5b411222

[valyala]

VictoriaMetrics takes into account the previous point before the range window in square brackets for implementing all the PromQL functions accepting range vectors, including rate, while Prometheus ignores the previous point before the range window. This resolves the following Prometheus issues:

• Graph disappears when step is smaller than 2xscrape_interval
• Increase over integer counter returns fractional value
• Improper rate() values

This may result in slightly different graphs between Prometheus and VictoriaMetrics like on the screenshots. Try increasing range window in square brackets - for example, use [10m] instead of [1m]. This should reduce the discrepancy between Prometheus and VictoriaMetrics graphs.

[valyala]

Closing this issue as resolved. Feel free opening new one if significant discrepancies between VictoriaMetrics graphs and Prometheus graphs are detected.