[Docs][DocDB] Statistics-based full compactions documentation #17667
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for infallible-bardeen-164bc9 ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
ddhodge
approved these changes
Jun 13, 2023
| @@ -47,6 +47,16 @@ In addition to throttling controls for compactions, YugabyteDB does a variety of | |||
|
|
|||
| YugabyteDB allows compactions to be externally triggered on a table using the [`compact_table`](../../../admin/yb-admin/#compact-table) command in the [yb-admin utility](../../../admin/yb-admin/). This is useful when new data is no longer coming into the system for a table and you might want to reclaim disk space due to overwrites or deletes that have already happened, or due to TTL expiry. | |||
|
|
|||
| ### Statistics-based full compactions to improve read performance | |||
|
|
|||
| YugabyteDB tracks the number of key/value pairs that are read at the DocDB level over a sliding period of time (dectated by the [`auto_compact_stat_window_seconds](../../reference/configuration/yb-tserver.md#auto_compact_stat_window_seconds) gflag). If it is detected that an overwhelming amount of the DocDB reads in a tablet is spent skipping over tombstoned and obsolete keys, then a full compaction will be triggered to remove the unnecessary keys. | |||
There was a problem hiding this comment.
Suggested change
| YugabyteDB tracks the number of key/value pairs that are read at the DocDB level over a sliding period of time (dectated by the [`auto_compact_stat_window_seconds](../../reference/configuration/yb-tserver.md#auto_compact_stat_window_seconds) gflag). If it is detected that an overwhelming amount of the DocDB reads in a tablet is spent skipping over tombstoned and obsolete keys, then a full compaction will be triggered to remove the unnecessary keys. | |
| YugabyteDB tracks the number of key-value pairs that are read at the DocDB level over a sliding period of time (dictated by the [`auto_compact_stat_window_seconds](../../../reference/configuration/yb-tserver/#auto_compact_stat_window_seconds) YB-Tserver flag). If YugabyteDB detects an overwhelming amount of the DocDB reads in a tablet are skipping over tombstoned and obsolete keys, then a full compaction will be triggered to remove the unnecessary keys. |
|
|
||
| YugabyteDB tracks the number of key/value pairs that are read at the DocDB level over a sliding period of time (dectated by the [`auto_compact_stat_window_seconds](../../reference/configuration/yb-tserver.md#auto_compact_stat_window_seconds) gflag). If it is detected that an overwhelming amount of the DocDB reads in a tablet is spent skipping over tombstoned and obsolete keys, then a full compaction will be triggered to remove the unnecessary keys. | ||
|
|
||
| Once all of the following conditions are met, an full compaction will automatically be triggered on the tablet. Those conditions include: the ratio of obsolete (e.g. deleted or removed due to TTL) versus active keys read reaches a threshold [`auto_compact_percent_obsolete`](../../reference/configuration/yb-tserver.md#auto_compact_percent_obsolete), and enough keys have been read within the window(`[auto_compact_min_obsolete_keys_found`](../../reference/configuration/yb-tserver.md#auto_compact_min_obsolete_keys_found)). This feature is compatible with tables with TTL, but will not schedule compactions on tables with TTL if the [TTL file expiration](../../develop/learn/ttl-data-expiration-ycql/#efficient-data-expiration-for-ttl) feature is active. |
There was a problem hiding this comment.
Suggested change
| Once all of the following conditions are met, an full compaction will automatically be triggered on the tablet. Those conditions include: the ratio of obsolete (e.g. deleted or removed due to TTL) versus active keys read reaches a threshold [`auto_compact_percent_obsolete`](../../reference/configuration/yb-tserver.md#auto_compact_percent_obsolete), and enough keys have been read within the window(`[auto_compact_min_obsolete_keys_found`](../../reference/configuration/yb-tserver.md#auto_compact_min_obsolete_keys_found)). This feature is compatible with tables with TTL, but will not schedule compactions on tables with TTL if the [TTL file expiration](../../develop/learn/ttl-data-expiration-ycql/#efficient-data-expiration-for-ttl) feature is active. | |
| Once all of the following conditions are met, a full compaction is automatically triggered on the tablet: | |
| - The ratio of obsolete (for example, deleted or removed due to TTL) versus active keys read reaches the threshold [auto_compact_percent_obsolete](../../../reference/configuration/yb-tserver/#auto_compact_percent_obsolete). | |
| - Enough keys have been read within the window [auto_compact_min_obsolete_keys_found](../../../reference/configuration/yb-tserver/#auto_compact_min_obsolete_keys_found). | |
| While this feature is compatible with tables with TTL, YugabyteDB won't schedule compactions on tables with TTL if the [TTL file expiration](../../../develop/learn/ttl-data-expiration-ycql/#efficient-data-expiration-for-ttl) feature is active. |
|
|
||
| ### Scheduled full compactions | ||
|
|
||
| YugabyteDB allows full compactions over all data in a tablet to be scheduled automatically using the [`scheduled_full_compaction_frequency_hours`](../../reference/configuration/yb-tserver.md#scheduled_full_compaction_frequency_hours) and [`scheduled_full_compaction_jitter_factor_percentage`](../../reference/configuration/yb-tserver.md#scheduled_full_compaction_jitter_factor_percentage) gflags. This can be useful for performance and disk space reclamation for workloads with a large number of overwrites or deletes on a regular basis. Can be used with tables with TTL as well, but is not compatible with the [TTL file expiration](../../develop/learn/ttl-data-expiration-ycql/#efficient-data-expiration-for-ttl) feature. |
There was a problem hiding this comment.
Suggested change
| YugabyteDB allows full compactions over all data in a tablet to be scheduled automatically using the [`scheduled_full_compaction_frequency_hours`](../../reference/configuration/yb-tserver.md#scheduled_full_compaction_frequency_hours) and [`scheduled_full_compaction_jitter_factor_percentage`](../../reference/configuration/yb-tserver.md#scheduled_full_compaction_jitter_factor_percentage) gflags. This can be useful for performance and disk space reclamation for workloads with a large number of overwrites or deletes on a regular basis. Can be used with tables with TTL as well, but is not compatible with the [TTL file expiration](../../develop/learn/ttl-data-expiration-ycql/#efficient-data-expiration-for-ttl) feature. | |
| YugabyteDB allows full compactions over all data in a tablet to be scheduled automatically using the [scheduled_full_compaction_frequency_hours](../../../reference/configuration/yb-tserver/#scheduled_full_compaction_frequency_hours) and [scheduled_full_compaction_jitter_factor_percentage](../../../reference/configuration/yb-tserver/#scheduled_full_compaction_jitter_factor_percentage) YB-TServer flags. This can be useful for performance and disk space reclamation for workloads with a large number of overwrites or deletes on a regular basis. This can be used with tables with TTL as well, but is not compatible with the [TTL file expiration](../../../develop/learn/ttl-data-expiration-ycql/#efficient-data-expiration-for-ttl) feature. |
|
|
||
| ##### --scheduled_full_compaction_frequency_hours | ||
|
|
||
| The frequency with which full compactions should be scheduled on tablets. `0` indicates that the feature is disabled. Recommended value: `720` hours or greater (i.e. 30 days). |
There was a problem hiding this comment.
Suggested change
| The frequency with which full compactions should be scheduled on tablets. `0` indicates that the feature is disabled. Recommended value: `720` hours or greater (i.e. 30 days). | |
| The frequency with which full compactions should be scheduled on tablets. `0` indicates that the feature is disabled. Recommended value: `720` hours or greater (that is, 30 days). |
|
|
||
| Percentage of `scheduled_full_compaction_frequency_hours` to be used as jitter when determining full compaction schedule per tablet. Must be a value between `0` and `100`. Jitter is introduced to prevent many tablets from being scheduled for full compactions at the same time. | ||
|
|
||
| Jitter will be deterministically computed when scheduling a compaction, between 0 and (frequency * jitter factor) hours. Once computed, the jitter will be subtracted from the intended compaction frequency to determined the tablet's next compaction time. |
There was a problem hiding this comment.
Suggested change
| Jitter will be deterministically computed when scheduling a compaction, between 0 and (frequency * jitter factor) hours. Once computed, the jitter will be subtracted from the intended compaction frequency to determined the tablet's next compaction time. | |
| Jitter is deterministically computed when scheduling a compaction, between 0 and (frequency * jitter factor) hours. Once computed, the jitter is subtracted from the intended compaction frequency to determine the tablet's next compaction time. |
|
|
||
| The maximum number of post-split compaction tasks that can be queued simultaneously (compactions that remove irrelevant data from new tablets after splits). | ||
| Example: If `scheduled_full_compaction_frequency_hours` is `720` hours (i.e. 30 days), and `scheduled_full_compaction_jitter_factor_percentage` is `33` percent, each tablet will be scheduled for compaction every `482` hours to `720` hours. |
There was a problem hiding this comment.
Suggested change
| Example: If `scheduled_full_compaction_frequency_hours` is `720` hours (i.e. 30 days), and `scheduled_full_compaction_jitter_factor_percentage` is `33` percent, each tablet will be scheduled for compaction every `482` hours to `720` hours. | |
| Example: If `scheduled_full_compaction_frequency_hours` is `720` hours (that is, 30 days), and `scheduled_full_compaction_jitter_factor_percentage` is `33` percent, each tablet will be scheduled for compaction every `482` hours to `720` hours. |
jmeehan16
force-pushed
the
auto-compact-stats-doc
branch
from
0f05ae5
to
6aa292c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds statistics-based full compaction documentation to the
previewsection. This includes:yb-tserverpage, near the other compaction documentation.-- auto_compact_check_interval_sec
-- auto_compact_stat_window_seconds
-- auto_compact_percent_obsolete
-- auto_compact_min_obsolete_keys_found
-- auto_compact_min_wait_between_seconds
Also adds the Scheduled Full Compactions documentation to the
previewdocumentation (it already exists in thestabledocumentation).Addresses #15188