New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve composition #257

Closed
fra967 opened this Issue May 5, 2017 · 3 comments

Comments

Projects
2 participants
@fra967
Copy link
Member

fra967 commented May 5, 2017

For a field of type reference, currently API looks up every record in the relevant collection

Considering a collection with X fields of type reference, an API query for N records will turn into N * X + 1 DB queries.
In the case of multiple references, this grows exponentially.

However, the _ids for each referenced collection could be grouped into a single query, and composition could happen in memory, bringing down the total number of queries to X + 1, regardless of the multiplicity of the references.

@fra967

This comment has been minimized.

Copy link
Member Author

fra967 commented May 11, 2017

Let's consider the usual example of articles collection
which can reference multiple publications and categories, but also single author
so a reference could be a single string, or an array

        "_id" : ObjectId("587cb76880222c9e7266d469"),
        "author" : "5874b9a36898ce361e66f367",
        "categories" : [
                "587cb5ca80222c9e7266bb2e",
                "587cb5ca80222c9e7266bb2e",
                "587cb5ca80222c9e7266bb2e"
        ]
}

if you retrieve 10 articles, some may come from the same author, so when you build the list of 10 author ids ([1,2,3,2,2,4,5,6,7,3] some are duplicates; when you pass them to mongo in a $in query, mongo will return a smaller set (i.e. the unique authors [1,2,3,4,5,6,7], not necessarily in this order)
you probably need to sort them and make them unique before passing the query

@fra967

This comment has been minimized.

Copy link
Member Author

fra967 commented May 11, 2017

The same happens, but in a more complex way, with multiple reference ids like categories which could have the same id stored multiple times, as in the above example
Again you probably want to build a list of unique _ids, pass them in the $in query, and the resolve back the references

@jimlambie jimlambie added this to Backlog in API Version 2 May 31, 2017

@jimlambie jimlambie moved this from Backlog to In Progress in API Version 2 May 31, 2017

@jimlambie jimlambie modified the milestone: 2.0.0 May 31, 2017

@jimlambie jimlambie referenced this issue Jun 3, 2017

Merged

Version 2.0.0 #297

@jimlambie jimlambie moved this from In Progress to Review in API Version 2 Jun 3, 2017

@jimlambie

This comment has been minimized.

Copy link
Member

jimlambie commented Jun 3, 2017

Closed by d103f62

@jimlambie jimlambie closed this Jun 3, 2017

@jimlambie jimlambie removed the in progress label Jun 3, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment