Skip to content

fix retention worker on compressed chunks#130

Merged
Polliog merged 1 commit intodevelopfrom
fix/compression-retention
Feb 10, 2026
Merged

fix retention worker on compressed chunks#130
Polliog merged 1 commit intodevelopfrom
fix/compression-retention

Conversation

@Polliog
Copy link
Copy Markdown
Collaborator

@Polliog Polliog commented Feb 10, 2026

This pull request refactors the log retention cleanup logic to improve efficiency and reliability, especially when handling compressed chunks in TimescaleDB. The main change is the removal of explicit chunk decompression in favor of using TimescaleDB's native drop_chunks for maximum retention and a new batching strategy for deleting logs per day, which avoids decompressing chunks manually. The reporting interfaces and logic are updated accordingly.

Retention logic refactor and batching improvements:

  • Removed all logic related to decompressing chunks and the associated types, methods, and result fields, including findCompressedChunksToDecompress, decompressChunk, and all tracking of decompressed chunks. [1] [2] [3] [4] [5]
  • Added a new batchDeleteLogs method that deletes logs in daily time windows, which works efficiently with compressed chunks and avoids large transactions.

Retention execution flow changes:

  • Refactored executeRetentionForOrganization to use the new batching strategy, removing the need to decompress chunks before deleting logs. [1] [2]
  • Overhauled executeRetentionForAllOrganizations to:
    • Use drop_chunks for organizations with the maximum retention (no per-row deletes needed).
    • Group organizations by retention period and batch-delete logs for each group.
    • Report results per organization, even when deleting in groups.

API and reporting updates:

  • Removed all references to chunk decompression from the RetentionExecutionResult and RetentionExecutionSummary interfaces, as chunk decompression is no longer part of the process. [1] [2] [3] [4]
  • Added new reporting fields such as chunksDropped and retentionGroups to the summary logs for better observability of the new process.

These changes make the retention cleanup process safer, more scalable, and better aligned with TimescaleDB's best practices.

@Polliog Polliog merged commit 43b12f9 into develop Feb 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant