New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Auto-Compaction tool #727
Auto-Compaction tool #727
Conversation
We should tie it to an index version that the compacter can understand. |
closing while we investigate other methods of bringing the index sizes down |
For those visiting this PR as something to potentially make use of, rather than wire up the compactor via a decorator, we would now use a persistent bus listener, see how we do it with Lucene: https://github.com/juxt/crux/blob/master/crux-lucene/src/crux/lucene.clj |
This new module will compact documents older than certain valid-time 'water-mark'.
Documents are pruned when new documents arrive via
put
andcas
. The compaction operation happens as part of the main tx ingestion process.This is a basis on which to eventually build configurable policies, such as: