No datapoint from old datasources

[keyolk]

I'm trying to leverage Promxy as my unique endpoint while having Prometheus migration.
Seem it working nice if my datapoints just switched to the new datasource.
But not woking well if some metrics are dual written to both old and new datasources.

Let's say I have metric a

For migration, I make it dual write to the both datasource A, and B.
So B datasource only have some recent metrics.
and A have whole metrics.

but when I query through Promxy, I can't get old data which in A
Any clue why this happens?

[keyolk]

My configuration

      promxy:
        server_groups:
         - static_configs:
             - targets:
               - sigv4-proxy.victoria-metrics.svc.cluster.local.:8080
           path_prefix: '/workspaces/ws-224d07f9-9fc3-4b6f-9ab9-f1b5be37fbc1/'
           timeout: 1m
           absolute_time_range:
             end: '2024-06-01T00:00:00Z'
             truncate: true
         - static_configs:
             - targets:
               - o11y-victoria-metrics-cluster-vmselect.victoria-metrics.svc.cluster.local.:8481
           path_prefix: '/select/1/prometheus/'

           timeout: 1m
           query_params:
             nocache: 1
           absolute_time_range:
             start: '2024-06-01T00:00:00Z'
             truncate: true

Also tried with single servergroup

      promxy:
        server_groups:
         - static_configs:
             - targets:
               - sigv4-proxy.victoria-metrics.svc.cluster.local.:8080
               - o11y-victoria-metrics-cluster-vmselect.victoria-metrics.svc.cluster.local.:8481
           relabel_confgis:
             - source_labels: [__address__]
                action: replace
                target_label: __path_prefix
                replacement: '/workspaces/ws-224d07f9-9fc3-4b6f-9ab9-f1b5be37fbc1/'
                regex: .*8080.*
             - source_labels: [__address__]
                action: replace
                target_label: __path_prefix
                regex: .*8481.*
                replacement: '/select/1/prometheus/'
           timeout: 1m

[jacksontj]

First off; thanks for reaching out! I love to see people using the project -- and especially love a good puzzle (foreshadowing!) :D

So I was actually able to reproduce this issue with relative ease -- but debugging it down to the root cause took some time. To spare the details of my various theories and debugging (which took quite a few hours :D ) -- I figured out that this is due to how the TimeFilters were handling their segmentation (more details in #663).

Thankfully the fix here is easy; but I'm honestly a bit surprised we never noticed before. The TLDR here is it is VERY EASY to reproduce as long as your QueryRange step is larger than your lookback delta -- which generally needs a multi-day query. So a HUGE thanks for catching and reporting this issue!