[BUGFIX] tsdb/wlog.Checkpoint: Fix counting of histogram samples in stats.

[aknuds1]

I realized that histogram samples are wrongly counted in tsdb/wlog.Checkpoint, in that float samples are counted against stats.TotalSamples and stats.DroppedSamples respectively instead.

This PR rectifies the bug, and augments TestCheckpoint to verify that stats.TotalSamples and stats.DroppedSamples are correctly updated.

That said, I don't think any callers of Checkpoint use the stats return value. Should it be dropped?

[machine424]

Good catch!
It wasn't used even in the PR that introduced it prometheus-junkyard/tsdb#332, maybe it was meant for debugging/dev?
Maybe some external clients could be relying on it, it has been around for a long time.

[aknuds1]

@machine424 I was also thinking it's usedmeant for debugging/dev. Do you know if it's an omission that Checkpoint doesn't check record.FloatHistogramSamples? Looks to me as if it should, but I don't know what the consequences are.

[machine424]

Good question, I'm not familiar with that part of the code, I'll need to take a look.

[beorn7]

Maybe @jesusvazquez can help with your questions?

[beorn7]

I approve the fix, but let's hold back merging this until we have an idea if the stats can be ripped out entirely.

[bboreham]

Big picture, I want more stats about checkpoint. Quite often when I look at big memory usage, especially in Agent mode, it seems associated with the TSDB checkpoint, but it's hard to see what happened. Checkpoints only happen every 2 hours, and usually it takes days for the data to build up.

I guess I should try to write a PR that adds the information I would find useful.

[bwplotka]

(Hello from the bug scrub meeting!)

LGTM, and as per question if this should be removed, I think we have cases to utilize this more in the next PRs, so let's merge. Thanks! (also we need float histograms indeed)

[aknuds1]

also we need float histograms indeed

@bwplotka I can open another PR to count also float histograms then.

[bwplotka]

Totally

[aknuds1]

Made PR to make Checkpoint handle float histograms @bwplotka.