Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(journald source): Cursor/checkpoint fixes #1106

Merged
merged 2 commits into from Oct 29, 2019
Merged

fix(journald source): Cursor/checkpoint fixes #1106

merged 2 commits into from Oct 29, 2019

Commits on Oct 28, 2019

  1. Fix duplicated records after checkpoint restore in journald source 

    The checkpoint value we store for journald sources is the cursor for the
most recently output record. This means, when we seek back to it, the
next record to be sourced will be duplicated from the previous run.

Instead, we either need to advance to the next record immediately after
restoring the cursor or before saving the cursor. The former is less
complex to implement.

Signed-off-by: Bruce Guenter <bruce@untroubled.org>
    @bruceg
    bruceg committed Oct 28, 2019
    Configuration menu
    Copy the full SHA
    8670e0a View commit details
    Browse the repository at this point in the history

  2. Add batch option to the journald source 

    Previously, the journald source would not write out a checkpoint until
there were no new records left in the journal. In the event of an
interruptin during a long stretch of new records, this could cause
duplicated records.

This adds a batch limit to the reader so the checkpoint is updated at
least once per batch.

Signed-off-by: Bruce Guenter <bruce@untroubled.org>
    @bruceg
    bruceg committed Oct 28, 2019
    Configuration menu
    Copy the full SHA
    4950f32 View commit details
    Browse the repository at this point in the history