Add the include filter

[pulkitsinghal]

Notes to learn from mongodb's implementation:

1. https://github.com/strongloop/loopback-connector-mongodb/blob/367bc033a546130b9f834d17a9d4f769aaefef92/lib/mongodb.js#L730
2. https://github.com/strongloop/loopback-connector-mongodb/blob/2a90951c482508bdc06d04e5a5b87e8e54f1ae6f/test/imported.test.js#L9
3. loopback-datasource-juggler/test/include.test.js
   • https://github.com/strongloop/loopback-datasource-juggler/blob/master/test/include.test.js

In the meantime a workaround is to use the native filter instead of include to write the query using ES's DSL instead of loopback. Ofcourse you need to have already setup parent-child mappings in the datasource correctly no matter what.

[pulkitsinghal]

cc @anana54, @rabee3 and @zelphir - this issue may be of interest to you when its finished

[RabeaWahab]

thanks @pulkitsinghal, will be checking it.

[pulkitsinghal]

I'm thinking that this also ties into the finishing #24 first ... should I make the include filter smart enough to look at the mappings and then form queries differently to support both nested and parent/child approach?

Should I do the work in increments so that until posts for children #24 (in parent/child) aren't done ... only nested mappings are supported for includes.

Honestly I am not a fan of nested and like the parent/child approach more but someone who scales differently in production might disagree. So I would like to write the best generic implementation possible. Will have to give this some more thought.

[aquid]

@pulkitsinghal Please close this issue if you think the latest release serves our purpose. I know you want a better solution for include rather than using loopback's fallback way. Please refer to the commit above and do the needful.

[pulkitsinghal]

@aquid - How does the include implementation handle or behave in extremely large datasets? Can you or Bhushan share some numbers? If not that then maybe at least a high level prediction and reasoning if it will do well or poorly when 1k or 10k or 100k or 1m results are processed as part of a matching include query?