Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WAL Replay counter #3974

Merged
merged 19 commits into from
Jul 12, 2021
Merged

WAL Replay counter #3974

merged 19 commits into from
Jul 12, 2021

Conversation

owen-d
Copy link
Member

@owen-d owen-d commented Jul 8, 2021

This PR introduces an alternative method for skipping WAL entries during replay which have already been recovered via checkpoints. Previously, we took advantage of out of order errors to "skip" old logs until we saw new data. In preparation for removing the ordering constraint, we now include a per stream counter which:

  1. increments on every accepted entry
  2. resets after WAL recovery

This annotates WAL records with the counter value for their most recent entry and during replay the stream will skip any counter value less than or equal to it's own (which is recovered via checkpoint). This ensures that even once we enable out of order log ingestion, we won't duplicate writes that occur in both checkpoints and segments.

This includes the following metric (localhost example after a WAL recovery)

# HELP loki_ingester_wal_duplicate_entries_total Entries discarded during WAL replay due to existing in checkpoints.
# TYPE loki_ingester_wal_duplicate_entries_total counter
loki_ingester_wal_duplicate_entries_total 656

Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@owen-d
Copy link
Member Author

owen-d commented Jul 12, 2021

ref #1544

@owen-d owen-d merged commit 05b8537 into grafana:main Jul 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants