You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Virtually every database interface in Papiea seems to have some sort of list() method which fetches all matching entities and returns an array. This will not scale beyond small database sizes because we will run out of memory very quickly.
All of these APIs need to be replaced with APIs that return async iterators, so we never load the entire result set into memory at once.
The text was updated successfully, but these errors were encountered:
BTW, this has already caused a severe out-of-memory situation in Papiea right now, where if we have too many intent watchers, we cannot process them because there are places where we try to load the entire list into memory.
Worse, this is done at startup in the diff_resolver process, so the diff_resolver process almost immediately crashes and we cannot make progress on resolving any diffs. So mitigating this is a blocker issue right now (although a full fix can come later).
Virtually every database interface in Papiea seems to have some sort of list() method which fetches all matching entities and returns an array. This will not scale beyond small database sizes because we will run out of memory very quickly.
All of these APIs need to be replaced with APIs that return async iterators, so we never load the entire result set into memory at once.
The text was updated successfully, but these errors were encountered: