Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvements to Feeds #786

Merged
merged 13 commits into from
Jun 28, 2023
Merged

Performance improvements to Feeds #786

merged 13 commits into from
Jun 28, 2023

Conversation

Half-Shot
Copy link
Contributor

@Half-Shot Half-Shot commented Jun 22, 2023

Fixes #789

This PR does several important perf fixes to the feed reader system. In particular we're trying to fix memory / storage consumption. To achieve this, the PR does two things:

  • We drop the account data used by feed reader entirely, instead using Redis. This avoids having to store things in Hookshot's memory, and doing expensive load/save operations. Deployments of Hookshot using an in-memory storage provider will always initial sync on startup. This is a regression, but in practice it only means you miss items that were published during downtime. Anyone wishing for better reliability should be using Redis.
  • We now read the feeds from Rust, avoiding the memory cost of shifting potentially huge blocks of XML across the native module boundary.

Copy link
Contributor

@tadzik tadzik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some concerns, but broadly okay

src/Stores/MemoryStorageProvider.ts Outdated Show resolved Hide resolved
src/Stores/RedisStorageProvider.ts Outdated Show resolved Hide resolved
src/feeds/FeedReader.ts Show resolved Hide resolved
src/feeds/FeedReader.ts Outdated Show resolved Hide resolved
Copy link
Contributor

@tadzik tadzik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still some nitpicks, but I approve.

changelog.d/786.bugfix Outdated Show resolved Hide resolved
@Half-Shot Half-Shot marked this pull request as ready for review June 28, 2023 15:17
@Half-Shot Half-Shot requested a review from a team as a code owner June 28, 2023 15:17
@Half-Shot Half-Shot enabled auto-merge June 28, 2023 15:17
@Half-Shot Half-Shot added this pull request to the merge queue Jun 28, 2023
Merged via the queue into main with commit 3217b9e Jun 28, 2023
12 checks passed
@Half-Shot Half-Shot deleted the hs/feeds-redis branch June 28, 2023 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feed reader has unaccounted memory leak
2 participants