DB Autovacuum, and Autoanalyze thresholds - #508
Closed
illume wants to merge 1 commit into
Closed
Conversation
illume
force-pushed
the
autovac-thresholds
branch
2 times, most recently
from
November 3, 2021 12:35
20e4e87 to
f6695e5
Compare
illume
marked this pull request as ready for review
November 16, 2021 08:32
illume
force-pushed
the
autovac-thresholds
branch
from
November 16, 2021 08:45
f6695e5 to
8256231
Compare
Because autovacuum and autoanalyze are effectively disabled when tables are very large. This is because the default is 20% of a table (and 10% of a table for analyze). Instead we change it to when about 5,000 rows change. This value was chosen based on getting the autovacuum to run every day. Because it's large enough to not cause the autovac to run all the time, but about the right size to make a difference for query statistics and reducing table bloat. The analyze threshold was chosen at half the autovacuum threshold because the defaults are set at half.
illume
force-pushed
the
autovac-thresholds
branch
from
November 16, 2021 08:49
8256231 to
344814a
Compare
Contributor
Author
|
The lint fail is because the proof-of-concept branch is failing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This hopefully addresses the DB table/index bloat size issue #507
Because autovacuum and autoanalyze are effectively disabled when tables
are very large. This is because the default is 20% of a table (and 10%
of a table for analyze).
Instead we change it to when about 5,000 rows change. This value was chosen
based on getting the autovacuum to run every day. Because it's large enough
to not cause the autovac to run all the time, but about the right size
to make a difference for query statistics and reducing table bloat.
The analyze threshold was chosen at half the autovacuum threshold
because the defaults are set at half.
Ultimately there should be some sort of auto-tuning done where the table growth/change rates are calculated and then these thresholds are updated. It has happened where a table gets a 5x growth over other months inside one month. It could be even greater. If it's only 10x growth then the autovac would only run 10 times per day, which is acceptable.
Testing done
Ran some tests to reproduce index bloat without the thresholds. #507 (comment) Then confirmed that table bloat was almost reduced entirely, auto analyze was running and index bloat was reduced to 2x compared to 10x observed.