spooling is broken

[Dieterbe]

endpoint went down, we spool to disk, endpoint comes back up, we unspool, but not correctly: some data gets lost as can be seen.
there's usually an interleaved pattern of data that makes it back, as can be seen here.
sometimes preceeded by a period of full data loss, sometimes not.

[Dieterbe]

should be fixed in master, will verify and update this

[schainker-digbil]

This does not appear to be working in master. This is a bit of a dumb question, but should we be making sure the metrics also have timestamps in order for them to spool back properly?

[Dieterbe]

should we be making sure the metrics also have timestamps in order for them to spool back properly?

the metrics are all <key> <val> <timestamp> triplets, that's how they are sent via carbon so that's how they are stored and replayed. no issues here :)

[Dieterbe]

you're right.
for some reason when the dest comes back up it doesn't process anything for the dest.

btw the git repo now has a grafana dashboard which makes it easy to see what's going on.
since cf6500c

[Dieterbe]

so the last visible issue was very much a metering problem.
that is fixed now

however, there is another bug somewhere preventing metrics from properly arriving (whether after spooling or not), see #45, so we must first fix that. once that's done, we can verify spooling

[Dieterbe]

since #45 is fixed (actually 45 was due to me dropping data before I sent it into the relay.. oops)
I confirmed this again. verified carbon-relay-ng's internal metrics, as well as dashboards to make sure all data displays correctly.