Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allowing partial commits of Segments being batch compacted. #23

Closed
simerplaha opened this issue Oct 19, 2018 · 2 comments
Closed

Allowing partial commits of Segments being batch compacted. #23

simerplaha opened this issue Oct 19, 2018 · 2 comments

Comments

@simerplaha
Copy link
Owner

Currently as Segments are batch compacted from one Level to another a failure to commit any one Segment will revert the entire compaction. The compaction is retried again based on the throttle configuration.

These failure can occur if a Segment is closed(maxSegmentsOpen) while its being read.

To avoid retries and to save IO partial commits of the batched Segments should be allowed - to write as many Segments as possible and alert the upper Level of the partial commit where only the failed Segments will be retried.

@simerplaha
Copy link
Owner Author

Fixing this issue will be a minor improvement to the compaction as these failures can be avoided entirely by giving maxSegmentsOpen a large number.

@simerplaha
Copy link
Owner Author

Commits can now be spread to multiple lower levels to be considered full committed. Issue #11 basically resolves this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant