Harden Closing Behavior

[matttproud]

Upon close or close request, …

1. Prometheus should go into a drain mode immediately whereby no further retrievals or queries are answered.
2. Once in drain mode, it should flush all pending metrics for appending into the storage infrastructure.
3. The storage infrastructure, should then begin a flush procedure of its own—e.g., moving in-memory values to the on-disk LevelDB store. After this step is finished, we should be safe to shut down.

Item no. 3 is pertinent now, as it is possible, though unlikely, to introduce metric index corruption for which we do not have any tools to perform referential integrity checks on the LevelDB storage. Quick example:

1. Metric and sample are requested to be appended.
2. LevelDB storage checks indices for metric.
3. No index element is found; it creates a preliminary one for label name and label value pairs.
4. Finally an index is made for the entire metric.
5. Sample is stored.

The ordering for no. 2 and no. 3 may be wrong; but although this mutation process is idempotent, we would never set the fingerprints for the new metric correctly.

Proposal:

1. Fix the problem as I described above.
2. Create an offline referential integrity scanner and repair utility. This would not take too long to due and would simply require the LevelDB iterator type and the model decoders.

[juliusv]

Closing since this is obsolete/fixed with the new storage.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.