Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvements when counting inverse collections #1700

Open
alcaeus opened this issue Dec 21, 2017 · 3 comments
Open

Performance improvements when counting inverse collections #1700

alcaeus opened this issue Dec 21, 2017 · 3 comments
Labels
Projects

Comments

@alcaeus
Copy link
Member

alcaeus commented Dec 21, 2017

This was previously handled in #1086 for 1.x, but can't be done in 2.0 for repository methods (they are expected to return a cursor which no longer offers triggering a count query).

While this performance improvement can lead to situations where counting gives one results and a subsequent iteration over the collection yields more results than previously counted, it is definitely not a good idea to read (and hydrate) all results just to count them.

A workaround may be to load data into memory but not hydrate it until iteration starts. This would eliminate the time penalty of hydration when counting a collection, but may still cause large memory consumption when counting a large collection.

@malarzm
Copy link
Member

malarzm commented Mar 14, 2018

Given repository method is to return an array of objects it might be hard to defer hydration. I'm inclined to move this into 2.x for now

@malarzm malarzm modified the milestones: 2.0.0, 2.x Mar 14, 2018
@alcaeus
Copy link
Member Author

alcaeus commented Mar 14, 2018

In 1.x, repository method is expected to return a cursor, not an array of objects. The problem in 2.0 arises because a cursor can no longer be counted, so we'd have to take the cursor, get its query information and run the appropriate count query. Personally, the memory issue is a big problem, as the only other option to solve this is to manually keep track of the number of associated referenced items.

@malarzm
Copy link
Member

malarzm commented Mar 14, 2018

That actually should be doable, PersistentCollection would need to have its count() method tweaked.

@alcaeus alcaeus added this to 2.0 in Roadmap Jul 13, 2018
@alcaeus alcaeus moved this from 2.0 to 2.1 in Roadmap Jan 15, 2019
@alcaeus alcaeus removed this from To Do in ODM 2.0 Feb 12, 2019
@alcaeus alcaeus changed the title [2.0] Performance improvements when counting inverse collections Performance improvements when counting inverse collections Feb 12, 2019
@alcaeus alcaeus moved this from 2.1 to 2.x in Roadmap Nov 5, 2019
@alcaeus alcaeus removed this from the 2.x milestone Aug 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
Roadmap
2.x (future scope)
Development

No branches or pull requests

2 participants