Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] Add database maintenance section; update info message on ANALYZE run (sqlite) #2756

Merged
merged 5 commits into from
Mar 14, 2024

Conversation

tsmethurst
Copy link
Contributor

@tsmethurst tsmethurst commented Mar 13, 2024

Description

If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.

If this is a documentation change, please briefly describe what you've changed and why.

The full ANAYLZE command we run for SQLite after each DB migration is not really necessary, and takes bloody ages (minutes and minutes on my 1cpu 1gb ram machine). This PR updates the post-migration ANALYZE command to use an analysis limit of 10000, which should be plenty (for comparison, we use 1000 for on-close callbacks).

Correction to the above: I've been messing about with running partial and full analyzes, and it turns out that running PRAGMA anaylsis_limit=10000; ANALYZE; actually doesn't cut it for the purposes of helping the query optimizer use the correct indexes.

Here's the contents of sqlite_stat1 for statuses after a full ANALYZE:

image

And here's the same after a partial ANALYZE:

image

Huge difference in emphasis across some of the important indexes.

Running the statuses web view query after a full analyze uses the intended index:

image

Where running the same query after a partial analyze causes the query planner to not use correct indexes:

image

So in other words, we should actually keep that full analyze after all!

Also adds some v. basic docs about db maintenance, for SQLite at least.

Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: [ ] -> [x]

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

  • I/we have read the GoToSocial contribution guidelines.
  • I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
  • I/we have not leveraged AI to create the proposed changes.
  • I/we have performed a self-review of added code.
  • I/we have written code that is legible and maintainable by others.
  • I/we have commented the added code, particularly in hard-to-understand areas.
  • I/we have made any necessary changes to documentation.
  • I/we have added tests that cover new code.
  • I/we have run tests and they pass locally with the changes.
  • I/we have run go fmt ./... and golangci-lint run.

@tsmethurst tsmethurst marked this pull request as draft March 14, 2024 10:08
@NyaaaWhatsUpDoc
Copy link
Member

if you're now going to not limit the size of the analyze run and instead using this for documentation changes, i think it'd be worth renaming the PR

@tsmethurst
Copy link
Contributor Author

Will do yeah :)

@tsmethurst tsmethurst changed the title [chore] Limit size of ANALYZE run after migration (sqlite) [docs] Add database maintenance section; update info message on ANALYZE run (sqlite) Mar 14, 2024
@tsmethurst tsmethurst marked this pull request as ready for review March 14, 2024 16:47
internal/db/bundb/bundb.go Outdated Show resolved Hide resolved
docs/admin/database_maintenance.md Outdated Show resolved Hide resolved
If this is the case, you can try manually running a full `ANALYZE` command, by doing the following:

1. Stop GoToSocial.
2. While connected to your GoToSocial database file in the `sqlite3` shell, run `PRAGMA analysis_limit=0; ANALYZE;` (this may take quite a few minutes).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2. While connected to your GoToSocial database file in the `sqlite3` shell, run `PRAGMA analysis_limit=0; ANALYZE;` (this may take quite a few minutes).
2. While connected to your GoToSocial database file in the `sqlite3` shell, run `ANALYZE;` (this may take quite a few minutes).

There's not much point to PRAGMA analysys_limit=0, it's the same thing as not setting it at all.

Co-authored-by: Daenney <daenney@users.noreply.github.com>
@NyaaaWhatsUpDoc NyaaaWhatsUpDoc merged commit 9b94b1f into main Mar 14, 2024
3 checks passed
@NyaaaWhatsUpDoc NyaaaWhatsUpDoc deleted the sqlite_analyze_limit branch March 14, 2024 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants