Query without timestamp sometimes returns empty result

[ctrox]

With promxy version v0.0.44 and up I sometimes get an empty result when querying promxy like this:

$ curl "localhost:8082/api/v1/query?query=up"
{"status":"success","data":{"resultType":"vector","result":[]}}

As soon as I set the time request parameter the issue disappears. I was unable to gather anything interesting from the promxy debug logs, here we have two requests, the first one worked and the second one produced an empty result (see the differences in the response size)

time="2019-08-29T14:11:00Z" level=debug msg="http://promxy-backend-2-wcpvc1:9090" api=Query query=up ts="2019-08-29 14:11:00.05469855 +0000 UTC m=+2944.129477951"
time="2019-08-29T14:11:00Z" level=debug msg="http://promxy-backend-1-wcpvc1:9090" api=Query query=up ts="2019-08-29 14:11:00.05469855 +0000 UTC m=+2944.129477951"
time="2019-08-29T14:11:00Z" level=debug msg="http://promxy-backend-1-wcpvc1:9090" api=Query query=up took=11.287815ms ts="2019-08-29 14:11:00.05469855 +0000 UTC m=+2944.129477951"
time="2019-08-29T14:11:00Z" level=debug msg="http://promxy-backend-2-wcpvc1:9090" api=Query query=up took=12.662061ms ts="2019-08-29 14:11:00.05469855 +0000 UTC m=+2944.129477951"
172.16.0.18 - - [29/Aug/2019 14:11:00] "GET /api/v1/query HTTP/1.1 200 63" 0.020217 query=up
time="2019-08-29T14:11:00Z" level=debug msg="http://promxy-backend-2-wcpvc1:9090" api=Query query=up ts="2019-08-29 14:11:00.720299096 +0000 UTC m=+2944.795078501"
time="2019-08-29T14:11:00Z" level=debug msg="http://promxy-backend-1-wcpvc1:9090" api=Query query=up ts="2019-08-29 14:11:00.720299096 +0000 UTC m=+2944.795078501"
time="2019-08-29T14:11:00Z" level=debug msg="http://promxy-backend-1-wcpvc1:9090" api=Query query=up took=6.673307ms ts="2019-08-29 14:11:00.720299096 +0000 UTC m=+2944.795078501"
time="2019-08-29T14:11:00Z" level=debug msg="http://promxy-backend-2-wcpvc1:9090" api=Query query=up took=13.621143ms ts="2019-08-29 14:11:00.720299096 +0000 UTC m=+2944.795078501"
172.16.0.18 - - [29/Aug/2019 14:11:00] "GET /api/v1/query HTTP/1.1 200 23011" 0.018809 query=up

I'm running promxy in front of two prometheus v2.11.1 instances on kubernetes. If I downgrade to v0.0.43 I cannot reproduce the issue. Anything later produces empty results in about half of the requests.

[jacksontj]

Thanks for the bug report! Sure enough I can reproduce this issue with ease. I was able to debug the issue and found that it is actually a prometheus issue (prometheus/prometheus#5972) TLDR the parseTime function on the prometheus side rounds the time -- which is causing the issues. I have an issue open upstream to get that resolved.

While that gets sorted out I have implemented a workaround in #213