Duplicate Traces created on Ingester crash/restart

[joe-elliott]

The ingester is designed to keep completed/flushed blocks for a configurable time period after it has been flushed. This time period exists to allow the queriers to pull new blocklists successfully before an ingester "forgets" a trace as well as to provide faster lookups on recent traces.

However, these completed blocks are stored in the wal folder. If an ingester restarts it will replay all of these traces locally and re-flush them to the backend.

[joe-elliott]

Proposal:
After a block is flushed to the backend move it to a folder named flushed beneath the wal folder. On startup clear the flushed folder.