Handle null values in Range Partition dimension distribution

[kfaraz]

Description

This PR adds support for handling null dimension values while creating partition boundaries
in range partitioning.

This means that we can now have partition boundaries like [null, "abc"] or ["abc", null, "def"].

This fixes the following problems seen in production environments.

• In cases of very sparse data, where there is no row that contains non-null values
  for all partition dimensions, partitioning just doesn't work. This is because all the rows
  are ignored in the PartialDimensionDistributionTask and intervalToPartitions turns
  out to be empty.
• If there are too many rows where the first partition dimension is null, all of them would
  end up in the first partition thus bloating up that partition. This can now be mitigated in
  cases where the later dimensions have non-null values which vary over these rows.

The tuple of nulls

Please note that even though [null, null] is also a valid boundary now, it will never actually
be encountered in practice. This is because lexicographically, the null tuple is
immediately followed the tuple of nulls. i.e. null < [null, null]
Thus, if the start and end of a partition were null and [null, null] respectively,
that partition would contain zero rows (end is not inclusive). There would be no
point in having such a partition and so it never occurs in practice.

Changes

• Do not skip nulls in PartialDimensionDistributionTask
• Add a null-safe serde to use instead of ArrayOfStringsSerde

Note:
The null-safe impl of ArrayOfStringsSerde can be removed once it is contributed
back to apache-datasketches itself.

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[abhishekagarwal87]

can this be selectively turned on only when more than one dimension is being used? I don't know for certain what the impact of not skipping null will be but then that impact will be limited to new range partitioning only. or it can be based on a flag that you can pass via the context. thoughts?

[kfaraz]

I think we should be fine without the flag.

The following effects would be observed on single dim:

1. Partitioning would now also work on a dimension column that is always null, although it will actually create just one partition.
2. Estimation of partition boundaries will also take into account null values. So the algorithm would do a better job of estimating the size of the first partition (it would be closer to the target rows). The sizes of later partitions will not be affected (although the same data being ingested before and after this change could have different partition boundaries as the first partition boundary might shift and the others would shift with it)

With the addition of (multi dimension) range partitioning, single dim is inevitably being affected as it now goes through the multi dim flow itself. So this would only be another part of that overall effect.

[abhishekagarwal87]

Thank you @kfaraz. I have merged this change. I just realized that this new behaviour should be called out in the docs as well. will that be done in a follow-up PR?

[kfaraz]

Thanks, @abhishekagarwal87 . The doc changes for multi-dim partitioning are being made in #11983 . I will create a follow up PR that documents the change in this PR.