Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto-Compaction tool #727

Closed
wants to merge 21 commits into from
Closed

Auto-Compaction tool #727

wants to merge 21 commits into from

Conversation

jonpither
Copy link
Contributor

@jonpither jonpither commented Mar 12, 2020

This new module will compact documents older than certain valid-time 'water-mark'.

Documents are pruned when new documents arrive via put and cas. The compaction operation happens as part of the main tx ingestion process.

This is a basis on which to eventually build configurable policies, such as:

  1. reduce granularity of valid time histories older than 12 months to 1-day
  2. compact transaction time histories older than 3 months (useful for a correction-heavy scenario)

@jonpither jonpither added enhancement New feature or request ingest labels Mar 12, 2020
@jonpither jonpither added this to Awaiting Merge in XTDB Development via automation Mar 12, 2020
@jonpither jonpither added this to the Beta milestone Mar 12, 2020
@jonpither jonpither requested a review from jarohen March 23, 2020 11:09
@jarohen jarohen added optimising-disk-space-465 and removed enhancement New feature or request ingest labels Mar 24, 2020
@jarohen jarohen removed this from the Beta milestone Mar 24, 2020
@jarohen jarohen moved this from Awaiting Merge to In progress in XTDB Development Mar 25, 2020
@jonpither
Copy link
Contributor Author

We should tie it to an index version that the compacter can understand.

@jarohen
Copy link
Member

jarohen commented Mar 30, 2020

closing while we investigate other methods of bringing the index sizes down

@jarohen jarohen closed this Mar 30, 2020
@jarohen jarohen removed this from In progress in XTDB Development Mar 30, 2020
@jonpither jonpither mentioned this pull request Apr 7, 2020
@jonpither jonpither deleted the jp/compactor branch April 29, 2020 11:41
@jonpither
Copy link
Contributor Author

jonpither commented Nov 26, 2020

For those visiting this PR as something to potentially make use of, rather than wire up the compactor via a decorator, we would now use a persistent bus listener, see how we do it with Lucene: https://github.com/juxt/crux/blob/master/crux-lucene/src/crux/lucene.clj

@jarohen jarohen added the 1.x label Apr 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants