filter and sort by field of related collection

[hallettj]

Filter and sort by fields of a related collection.

This involves a change where relationship references can be "unified". We might have multiple references to the same relationship - for example one that selects fields, another that filters. When these are registered they are unified if they have the same key in the query request collection_relationships map, and they don't have incompatibilities such as differing predicates or offsets. Unifying involves merging field/column selections so that we can get necessary data with a single $lookup.

Previously we blindly copied documents from relationship $lookup pipelines to top-level row sets. But unification means that relationship pipelines may produce fields that were not requested by query request fields or aggregates. This change updates $replaceWith stages to prune data coming from $lookup pipelines to select only requested data.

The changes here are mostly to mongodb-specific code, with only small adjustments to the database-agnostic logic in ndc-query-plan.

Tickets:

• MDB-14
• MDB-15
• MDB-127

[hallettj]

I tried testing the the latest engine version, and found that the integration tests I had didn't pass because the new engine version uses Exists in cases where the old version used BinaryComparisonOperator. I pushed more commits to update the engine version the test config uses, and I fixed up the connector's handling of Exists expressions.

[dmoverton]

I think this module might be a good candidate for some property tests.

[dmoverton]

Looks good, but please consider adding some property tests or unit tests for unify_relationship_references.rs.

[codedmart]

👍

[hallettj]

Looks good, but please consider adding some property tests or unit tests for unify_relationship_references.rs.

You're right, I added some tests