Prometheus returns incorrect data with offset queries

[lesovsky]

Prometheus returns incorrect data when using offset queries

I'm querying metrics with offset and it looks like it's not working as expected. In the example below, I executed two queries, with offset and without. Returned metrics' timestamps differ by 1 second instead of 1 hour.

promtool query instant http://127.0.0.1:9090 'process_start_time_seconds{instance="prometheus:9090"} offset 1h' && sleep 1 && promtool query instant http://127.0.0.1:9090 'process_start_time_seconds{instance="prometheus:9090"}'
process_start_time_seconds{instance="prometheus:9090", job="prometheus"} => 1553678784.36 @[1553836264.418]
process_start_time_seconds{instance="prometheus:9090", job="prometheus"} => 1553678784.36 @[1553836265.448]

Asking metrics for an interval, all work as expected. In exmaple below, I added [1m] range to "offset query" and got two timestamps. Comparing them with timestamp returned by non-offset query shows expected 1 hour distance.

promtool query instant http://127.0.0.1:9090 'process_start_time_seconds{instance="prometheus:9090"}[1m] offset 1h' && sleep 1 && promtool query instant http://127.0.0.1:9090 'process_start_time_seconds{instance="prometheus:9090"}'
process_start_time_seconds{instance="prometheus:9090", job="prometheus"} =>
1553678784.36 @[1553832654.985]
1553678784.36 @[1553832684.985]
process_start_time_seconds{instance="prometheus:9090", job="prometheus"} => 1553678784.36 @[1553836312.44]

Environment

• System information:
  Linux 4.18.0-041800-generic x86_64

• Prometheus version:

prometheus, version 2.7.1 (branch: HEAD, revision: 62e591f928ddf6b3468308b7ac1de1c63aa7fcf3)
  build user:       root@f9f82868fc43
  build date:       20190131-11:16:59
  go version:       go1.11.5

• Prometheus configuration file:

global:
  scrape_interval:     30s
  evaluation_interval: 30s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['prometheus:9090']

[lesovsky]

Perhaps, it shows a right metric's value, but has a skewed timestamp.

[brian-brazil]

The timestamp of a instant vector result is always that of the evaluation time.

[lesovsky]

Thanks for clarification.
Such behaviour is confusing a bit, so it would be nice to mention this in the documentation regarding offset modifier.