Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache federation blocklist #3486

Merged
merged 2 commits into from Jul 5, 2023
Merged

Cache federation blocklist #3486

merged 2 commits into from Jul 5, 2023

Conversation

Nutomic
Copy link
Member

@Nutomic Nutomic commented Jul 5, 2023

All incoming and outgoing federation actions read the blocklist/allowlist and slur filters multiple times. This causes a huge number of database reads if we hit the db directly. So we cache these values for a short time, which will already make a huge difference and ensures that changes take effect quickly.

From @phiresky:

I want to mention that the fetch_local_site_data is the third most expensive function in the code base and the first-most expensive actually needed function. I'd recommend either this be merged or I can create a minimal PR that just caches fetch_local_site_data for a few seconds.

The cache duration can be significantly reduced and still have huge impact. Even though this function takes only 1 ms it is called with a frequency of over 1000Hz on lemmy.world. A cache duration of 5 seconds would be perfectly fine and even 1s would be useful.

Nutomic added a commit that referenced this pull request Jul 5, 2023
/// multiple times. This causes a huge number of database reads if we hit the db directly. So we
/// cache these values for a short time, which will already make a huge difference and ensures that
/// changes take effect quickly.
const BLOCKLIST_CACHE_DURATION: Duration = Duration::from_secs(60);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can def live with this short of a cache.

@dessalines dessalines merged commit ebaf69b into main Jul 5, 2023
2 checks passed
@phiresky phiresky mentioned this pull request Jul 6, 2023
Nutomic added a commit that referenced this pull request Jul 13, 2023
* Check for dead federated instances (fixes #2221)

* move to apub crate, use timestamp

* make it compile

* clippy

* use moka to cache blocklists, dead instances, restore orig scheduled tasks

* remove leftover last_alive var

* error handling

* wip

* fix alive check for instances without nodeinfo, add coalesce

* clippy

* move federation blocklist cache to #3486

* unused deps
Nutomic added a commit to cetra3/lemmy that referenced this pull request Jul 19, 2023
* Check for dead federated instances (fixes LemmyNet#2221)

* move to apub crate, use timestamp

* make it compile

* clippy

* use moka to cache blocklists, dead instances, restore orig scheduled tasks

* remove leftover last_alive var

* error handling

* wip

* fix alive check for instances without nodeinfo, add coalesce

* clippy

* move federation blocklist cache to LemmyNet#3486

* unused deps
Nutomic added a commit that referenced this pull request Jul 21, 2023
* Check for dead federated instances (fixes #2221)

* move to apub crate, use timestamp

* make it compile

* clippy

* use moka to cache blocklists, dead instances, restore orig scheduled tasks

* remove leftover last_alive var

* error handling

* wip

* fix alive check for instances without nodeinfo, add coalesce

* clippy

* move federation blocklist cache to #3486

* unused deps
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants