feat(search): De-duplicate scale factors across entities

[iprentic]

We have scales, such as downweighting deprecated entities, that are applied on multiple indices. Previously, when we were querying only one index at a time, only one of these would be applied at once, but now that we query multiple indices together, we have no de-duplicating logic. Therefore, the scales are being applied multiple times. This change is to de-duplicate the scale factors.

Tested locally on quickstart (since our test entity registry does not have these set up)
function scores section before the change:

          "functions": [
            {
              "filter": {
                "match_all": {
                  "boost": 1
                }
              },
              "weight": 1
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "active": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 2
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "materialized": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            }
          ]

Functions section after the change:

          "functions": [
            {
              "filter": {
                "match_all": {
                  "boost": 1
                }
              },
              "weight": 1
            },
            {
              "filter": {
                "term": {
                  "deprecated": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            },
            {
              "filter": {
                "term": {
                  "active": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 2
            },
            {
              "filter": {
                "term": {
                  "materialized": {
                    "value": "true",
                    "boost": 1
                  }
                }
              },
              "weight": 0.5
            }
          ]

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[iprentic]

• Since this PR does affect ranking, I ran the golden tests locally, and they passed. I think the failure here is due to the common CI failures we have been seeing across PRs.

[RyanHolstien]

Hmm, can't this just be changed in the lambda that's currently in the flatMap call? As in instead of streaming across all of them just pull the first one and calculate the weight factor? Doing an intermediate collect complicates the readability and is relatively expensive.

[RyanHolstien]

To deduplicate equivalent values in the stream you can use distinct()

[iprentic]

Yeah, I can do that! This is how it's done above but I'll rewrite it to do that (and maybe the one above too)