Backfill from OpenMetrics format

[asquare14]

Related PR : #7586
Design Document here

[bwplotka]

Some initial comments.

Good job, thanks!

[beorn7]

Just saw that @bwplotka has submitted feedback. I was in the middle of it, but then the dev summit started. I'll just submit my comments so far without further adjusting. (They were mostly nitpicking anyway.) Will then do the rest of my review on top of @bwplotka .

@asquare14 assume that I agree with anything @bwplotka said unless I explicitly say something else. (o:

[beorn7]

OK, a few more points. Looks like the generally correct approach.

[beorn7]

Expanding on ideas about the command structure we just brainstormed during the sync meeting:

As discussed, the command should not be import (and neither backfill, as in #7675) but something that is both more appropriate and applies to both. Our best idea so far is create-blocks-from, with the next sub-command telling what (e.g. openmetrics, rules). I think we should also try to stay as faithful as possible to existing flags and args conventions in the promtool commands. For example, promtool tsdb (dump|list|analyze) all use an argument (called "db path") to define where the TSDB files are. Here and in #7675, we use a flag.

Backfilling from rules: promtool tsdb create-blocks-from rules [<flags>] [<db path>]
Backfilling from OM file: promtool tsdb create-blocks-from openmetrics [<flags>] [<OM file>] [<db path>]

As for the other tsdb commands, the default value for <db path> is defaultDBPath, i.e. data/`.

The flag common to both sub-commands is block-length (hidden, default 2h).

The rules sub-command has the additional flags url (server to evaluate against), start, end, step, rule-file (ExistingFiles(), i.e. repeatable).

Since we want to allow multiple rule files for rule backfilling, but only one input file for backfilling from a file, we use a flag for the former but an argument for the latter.

[bwplotka]

Small nits, overall Looks good! 💪

[beorn7]

Just some comment nits, otherwise this LGTM.

But please, other reviewers, could you quickly doublecheck if your concerns are all addressed?

[codesome]

Lots of nits with 1 small fix in the test :)

[codesome]

I was about to hit merge but found 2 bugs in the last sanity check :)

[codesome]

LGTM!

[pathcl]

@bwplotka hey there! Im trying to use what you described here: https://grafana.com/blog/2020/09/02/how-were-improving-backfill-methods-to-get-older-data-into-prometheus/ however I wonder what would be the way to import csv nowadays given all these pull request merged. I did noticed this: https://github.com/prometheus/prometheus/blob/main/docs/storage.md but can't find an explicit example of importing a csv file as it was described in your talk/demo session.

Thanks a lot !!

[beorn7]

There is no direct way of importing metrics data from CSV yet. As it turned out, CSV is really not very well suited for metrics data, which makes it hard to come up with a good generally useful approach. Having said that, it's still on the agenda, we just didn't do it as the first thing.

The linked design doc has a bunch of concerns about CSV in the discussion. Contributions are welcome, but best to be discussed via the community channels first. (Questions and discussions on a closed GH PR easily get lost.)