Speed up iter handles in private proxy db #418
Conversation
|
We don't allow database code in the proxies in Gramps core. I seem to remember that an approach discussed previously was to pass a filter to the iterators and/or get_handle methods which would allow the backend to apply optimisations where appropriate. |
|
Makes sense. For Web API, it's OK as a fix for now, but in core I can work on a PR to add an object property filter method that falls back to iter_objects. |
|
There were a lot of discussions related to this back in 2016 on the A couple of threads jump out at me: Many of our developers contributed to the discussion. I suggest that you read the archives and restart the conversation. I seem to remember that we were leaning towards passing a Filter object into the iterators. |
|
The Isotammi group invited review of how their Neo4j based website approached filtering ... noting the close relationship to the Gramps database and it being open source. https://gramps.discourse.group/t/collaborate-on-optimizing-a-new-custom-rule/3063/10 |
In a first effort to fix #417, this PR drastically reduces the time taken by the iter handle methods of Gramps' private proxy db. Speedups of a factor of 30 and more are possible.
The existing implementation executed a database query for each object just to check whether it has private: true in its raw data. But we have a database column "private" and can simply filter for it in a single query!
This obviously only works when the base DB is a DBAPI instance, so if it is not, we fall back to the existing implementation.
@Nick-Hall, I could backport this to Gramps core as well if you think it makes sense.