Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace all list_*() methods in database interfaces with async iterators #679

Open
joshua-berry-ntnx opened this issue Apr 16, 2021 · 2 comments
Assignees
Labels
backlog bug Something isn't working
Milestone

Comments

@joshua-berry-ntnx
Copy link
Contributor

Virtually every database interface in Papiea seems to have some sort of list() method which fetches all matching entities and returns an array. This will not scale beyond small database sizes because we will run out of memory very quickly.

All of these APIs need to be replaced with APIs that return async iterators, so we never load the entire result set into memory at once.

@joshua-berry-ntnx joshua-berry-ntnx added this to the Customer-1 M2 milestone Apr 16, 2021
@joshua-berry-ntnx
Copy link
Contributor Author

BTW, this has already caused a severe out-of-memory situation in Papiea right now, where if we have too many intent watchers, we cannot process them because there are places where we try to load the entire list into memory.

Worse, this is done at startup in the diff_resolver process, so the diff_resolver process almost immediately crashes and we cannot make progress on resolving any diffs. So mitigating this is a blocker issue right now (although a full fix can come later).

@Emulebest
Copy link
Contributor

Let's discuss the possible alternatives here for concrete use cases

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backlog bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants