Improve composition

[fra967]

For a field of type reference, currently API looks up every record in the relevant collection

Considering a collection with X fields of type reference, an API query for N records will turn into N * X + 1 DB queries.
In the case of multiple references, this grows exponentially.

However, the _ids for each referenced collection could be grouped into a single query, and composition could happen in memory, bringing down the total number of queries to X + 1, regardless of the multiplicity of the references.

[fra967]

Let's consider the usual example of articles collection
which can reference multiple publications and categories, but also single author
so a reference could be a single string, or an array

        "_id" : ObjectId("587cb76880222c9e7266d469"),
        "author" : "5874b9a36898ce361e66f367",
        "categories" : [
                "587cb5ca80222c9e7266bb2e",
                "587cb5ca80222c9e7266bb2e",
                "587cb5ca80222c9e7266bb2e"
        ]
}

if you retrieve 10 articles, some may come from the same author, so when you build the list of 10 author ids ([1,2,3,2,2,4,5,6,7,3] some are duplicates; when you pass them to mongo in a $in query, mongo will return a smaller set (i.e. the unique authors [1,2,3,4,5,6,7], not necessarily in this order)
you probably need to sort them and make them unique before passing the query

[fra967]

The same happens, but in a more complex way, with multiple reference ids like categories which could have the same id stored multiple times, as in the above example
Again you probably want to build a list of unique _ids, pass them in the $in query, and the resolve back the references

[jimlambie]

Closed by d103f62