Documentation for admin API is no good for deleting metrics

[kaytrance]

Documentation must specify what unix timestamp format must be used to avoid total data loss (as it happened in my case).

Documentation just say

end=<rfc3339 | unix_timestamp>: End timestamp. Optional and defaults to maximum possible time.

In my case I have specified timestamp of 2019-05-13 17:00 as 1557756000000, but apparently it needs to be without last 3 digits (so in seconds, not in milliseconds).

In result my data was totally wiped out using this query:

/api/v1/admin/tsdb/delete_series?end=1557756000000&match[]=up{job="test"}

Suggestion:

1. modify documentation and add that unix timestamp must be in form of seconds;
2. add check in prometheus to validate unix timestamp (by length, for example), return "not accepted" if unix timestamp was in incorrect form.

[brian-brazil]

Unix timestamps are in seconds generally, as is everything else in the API as documented on the page.

1557756000000 is also a perfectly valid unix timestamp.

[kaytrance]

yes, it is valued, but in my case it caused all data to be wiped out because, probably, prometheus expected it to be in seconds and since it was in milliseconds it converted it to something quite far in future, and together with end parameter supplied it deleted everything until this date, meaning everything I had. So this leaves a possibility of very unpleasant situation, and I was thinking it would be a good idea to create a issue that could save other people for potential data loss.

[dannyk81]

@kaytrance

1557756000000 (or 1557756000 in seconds format) is Monday, May 13, 2019 14:00:00 (GMT/UTC) and not 17:00:00

Prometheus uses UTC times, so perhaps you had more data deleted due not accounting for TZ differences?

I always use milliseconds and never had an issue.

[brian-brazil]

Looking at the docs, where we document the semantics of unix_timestamp we don't actually use the string unix_timestamp so it'd harder to find than it should be. Similarly with rfc3339.

[Harkishen-Singh]

Prometheus uses UTC times, so perhaps you had more data deleted due not accounting for TZ differences?

I guess, he means to say that all the data got deleted due to a large end value.

Unix timestamps are in seconds generally, as is everything else in the API as documented on the page.

I do agree with what @brian-brazil said, but I think we can have add check in prometheus to validate unix timestamp as a sanitary check.

[brian-brazil]

I think we can have add check in prometheus to validate unix timestamp as a sanitary check.

I don't see how that'd help here, the timestamp was valid.

[roidelapluie]

On 01 Sep 04:11, Brian Brazil wrote: > I think we can have add check in prometheus to validate unix timestamp as a sanitary check. I don't see how that'd help here, the timestamp was valid.

We could refuse timestamps higher than 10000000000 date -d @10000000000 Sat Nov 20 18:46:40 CET 2286 to prevent people who think it is ms for doing so. And we can revisit that in the year 2250.

…

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #5567 (comment)

-- (o- Julien Pivotto //\ Open-Source Consultant V_/_ Inuits - https://www.inuits.eu

[Harkishen-Singh]

We could refuse timestamps higher than 10000000000

that's what I meant as a sanitary check.