feat(data-modeling): implement automatic relationship inference algorithm COMPASS-9776

[gribnoysup]

This patch adds the logic for automatically discovering relationships between collections following the algorithm we outlined in the "Infer relationships" scope. This behavior is behind a feature flag until we add the toggle to disable the discovery to the schema generation UI, this is the next chunk of work for the project.

I added some collections to the sample_airbnb database in our teams "Cluster 0" cluster if you want to test it. If you create a data model for this namespace, it should identify relationships between listingsAndReview, reviewers, and reviews collections:

[Anemy]

Looking good, left a couple comments, no blockers.

There are a few places we could check the abort signal to fail bit quicker if we want but I don't think we need to go into that now, similarly if we know it's a view already we don't need to do index checks. We aren't special casing views yet in data modeling though.

[addaleax]

I don't know if it's just me, but my first thought reading this function is "this callback could have been a generator". Obviously not really all that different in the end, but mentioning it in case you also feel that that would be more idiomatic

[gribnoysup]

That's a good idea and worth doing, let me adjust this

[gribnoysup]

Refactored in 811d84e

[gribnoysup]

@addaleax @Anemy Thanks for the reviews! I'm rerunning some flaky tests, but otherwise I think I addressed all the feedback, do you want to take another look when you have a moment?

[addaleax]

Looks great! A few comments but basically LGTM 🙂

[addaleax]

Just for my own understanding, in what type of situation would index.name be false-y?

[gribnoysup]

I don't really know, that's the type we're getting directly from the driver, so I just assumed in some corner cases it might be possible and decided to account for that, can do a bit more digging 🙂

[addaleax]

Suggested change

	).filter((value) => {
	// They will be matching in a lot of cases, but the chances of both
	// local and foreign field in a relationship being _id are very slim, so
	// skipping
	return value[0] !== '_id';
	).filter((propPath) => {
	// The types of _id will be matching in a lot of cases, but the chances of both
	// local and foreign field in a relationship being _id are very slim, so
	// skipping
	return propPath[0] !== '_id';

or at least that's what we're trying to communicate here, right?

Unfortunately, I think this would be a valid pattern that we do care about, where _ids of multiple collections are references to each other as a way of reducing a single document's size (e.g. you have a busy collection with smaller docs and a larger collection with rarely-accessed and potentially large docs that could be referenced via $lookup if necessary)

I think it's fine to skip this for now but if we do, we probably want a TODO ticket to think a bit more about this situation

[gribnoysup]

Hmmm, actually I was really just assuming there would never be a case like that and was trying to optimise a bit, if you think it's a possible scenario I think I can just drop this filter, I don't think performance-wise it's a very big change

[gribnoysup]

Removed the logic in c2c98ff, thanks for bringing this up!