Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes db access and migrations as well as splitting into multiple processes #2

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

chapel
Copy link
Contributor

@chapel chapel commented Nov 16, 2024

Lots of changes, I didn't mean to stack it all in one PR, but here we go.

  • Only cache posts with a score > 10
  • Move migrations to separate migration folder with commands to run them
  • Split cron, feed, stream scripts into separate processes that run at the same time
    • This is to avoid the forced sync nature of some db actions, e.g. cron job cache read is slow and blocks everything in the same process
  • Move to async db access wherever we can, it isn't the most useful since some of the query commands are still sync

@chapel
Copy link
Contributor Author

chapel commented Nov 16, 2024

Note if you deploy this, the initial delete there is going to take some time. Not sure how best to go about it, if you're okay with a short bit of downtime, you could manually run it against the db.

Edit: added a migration that you can run while the server is still running so you can prune low score posts without taking everything down. Though as well the split process model might mitigate the slowness/issue with a large delete like that.

…ate instances, move db migrations to separate folder
@chapel chapel changed the title Prune posts less than 10 score and add some notes to readme about caching Refactor to use async db access, split work into three processes, move migration code to specific migration area Nov 17, 2024
@chapel chapel changed the title Refactor to use async db access, split work into three processes, move migration code to specific migration area Changes db access and migrations as well as splitting into multiple processes Nov 17, 2024
@hipstersmoothie
Copy link
Owner

@chapel will deploy during the night

@chapel
Copy link
Contributor Author

chapel commented Nov 25, 2024

I will say, it seems to have been working well all this time, not sure how big the databases are now, but from an end user perspective the feeds have felt generally snappy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants