Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Introduce translog size and age based retention policies #25147
This PR extends the TranslogDeletionPolicy to allow keeping the translog files longer than what is needed for recovery from lucene. Specifically, we allow specifying the total size of the files and their maximum age (i.e., keep up to 512MB but no longer than 3 hours). This will allow making ops based recoveries more common.
To achieve the above the translog files are extended to contain a timestamp, indicating when they were created.
Note that the default size and age still set to 0, maintaining current behavior. This is needed as the other components in the system are not yet ready for a longer translog retention. I will adapt those in follow up PRs.
Relates to #10708
jasontedor left a comment
For me I think the only question is whether or not we should fsync the metadata when we close a writer into a reader. I do not seeing this being a question that should hold up this PR. I also do not see this being a challenging thing to do, we already sync (without syncing metadata) when we close into a reader, so I think we can merely push a boolean down to change this. Again, separate PR! LGTM.