Skip to content

DB Autovacuum, and Autoanalyze thresholds - #508

Closed
illume wants to merge 1 commit into
proof-of-conceptfrom
autovac-thresholds
Closed

DB Autovacuum, and Autoanalyze thresholds#508
illume wants to merge 1 commit into
proof-of-conceptfrom
autovac-thresholds

Conversation

@illume

@illume illume commented Nov 3, 2021

Copy link
Copy Markdown
Contributor

This hopefully addresses the DB table/index bloat size issue #507

Because autovacuum and autoanalyze are effectively disabled when tables
are very large. This is because the default is 20% of a table (and 10%
of a table for analyze).

Instead we change it to when about 5,000 rows change. This value was chosen
based on getting the autovacuum to run every day. Because it's large enough
to not cause the autovac to run all the time, but about the right size
to make a difference for query statistics and reducing table bloat.

The analyze threshold was chosen at half the autovacuum threshold
because the defaults are set at half.

Ultimately there should be some sort of auto-tuning done where the table growth/change rates are calculated and then these thresholds are updated. It has happened where a table gets a 5x growth over other months inside one month. It could be even greater. If it's only 10x growth then the autovac would only run 10 times per day, which is acceptable.

Testing done

Ran some tests to reproduce index bloat without the thresholds. #507 (comment) Then confirmed that table bloat was almost reduced entirely, auto analyze was running and index bloat was reduced to 2x compared to 10x observed.

  • Check the table growth rates.
  • Check if there are other tables to add to this list. Only the tables from the cleanup script were chosen.

@illume illume mentioned this pull request Nov 3, 2021
11 tasks
@illume
illume requested a review from yolossn November 3, 2021 10:22
@illume
illume changed the base branch from main to proof-of-concept November 3, 2021 10:23
@illume
illume force-pushed the autovac-thresholds branch 2 times, most recently from 20e4e87 to f6695e5 Compare November 3, 2021 12:35
@illume
illume marked this pull request as ready for review November 16, 2021 08:32
@illume
illume force-pushed the autovac-thresholds branch from f6695e5 to 8256231 Compare November 16, 2021 08:45
@illume
illume changed the base branch from proof-of-concept to main November 16, 2021 08:45
Because autovacuum and autoanalyze are effectively disabled when tables
are very large. This is because the default is 20% of a table (and 10%
of a table for analyze).

Instead we change it to when about 5,000 rows change. This value was chosen
based on getting the autovacuum to run every day. Because it's large enough
to not cause the autovac to run all the time, but about the right size
to make a difference for query statistics and reducing table bloat.

The analyze threshold was chosen at half the autovacuum threshold
because the defaults are set at half.
@illume
illume force-pushed the autovac-thresholds branch from 8256231 to 344814a Compare November 16, 2021 08:49
@illume
illume changed the base branch from main to proof-of-concept November 16, 2021 08:49
@illume

illume commented Nov 16, 2021

Copy link
Copy Markdown
Contributor Author

The lint fail is because the proof-of-concept branch is failing make backend-code-check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant