Configure output plugin with checkpoint initialized to newest/oldest message in queue

[bbinet]

See #24.

If no checkpoint exists
e.g., initial_checkpoint = "oldest|newest" (defaulting to oldest, the current behaviour)

[bbinet]

Should we rather change the initial_checkpoint config option name so that it can be a boolean value (true/false)?
Something like initial_checkpoint_newest = true|false (defaulting to false)?

[bbinet]

e7376a3 adds support for initial_checkpoint_newest option for output plugins:
If no checkpoint exists, the checkpoint can now be initialized to the oldest/newest message with the following configuration:
initial_checkpoint_newest = true|false (defaulting to false)

[bbinet]

FYI, I've rebased this branch on the latest master branch

[trink]

The feature is fine but the implementation would have to change to play better with pruning (there are several ways this can happen but I haven't found anything I like yet) . I am heads down on the lua_sandbox work I need to get done so it is unlikely this will be addressed in Q2.

[bbinet]

Ok. Please tell me if I can be of any help.

[trink]

Proposal for Output Plugin Checkpoint Initialization

1. When starting Hindsight, output plugins with no checkpoint entry should have their value set to the beginning (oldest data in the queue)

• Pros
  • consistent with analysis plugins
  • easy to reason about (deterministic)
  • useful for testing/reporting (can quickly clear checkpoints and re-run data between executions)
• Cons
  • none; If different behavior is desired (starting from the end, starting from message 23, whatever) the checkpoint file can be hand crafted or generated by a utility in a deterministic way creating reproducible behavior.

2. When loading a dynamic output plugin with no checkpoint entry it should have its value set to the current location (end of the last message flushed to disk).

• Pros
  • consistent with analysis plugins
  • eliminates race conditions with the pruner
• Cons
  • Not really a con but historical data can only be processed with a Hindsight restart

Problems with the EOF checkpointing

• the checkpoint will most likely be improperly positioned; i.e., not at the start of a message (the reader properly handles this but you would get a discarded byte count back indicating corruption where there really isn't any)
• on startup it is non-deterministic where the plugin will actually start since the EOF can vary from run to run with each plugin startup time. For real-time this is a non-issue but it would not work in a test/debug scenario. We could make some changes to make it deterministic but at the moment there is no value/use case.

@bbinet @whd feedback?

[trink]

Superseded by PR #52

[bbinet]

@trink Your proposal makes complete sense, thanks.