Enable segments read/published stats on Compaction task completion reports

[adithyachakilam]

Description

This adds visibility into number of segments read/published by each parallel compaction task.

This PR adds new subclass for IngestionStatsAndErrorsTaskReportData to take new fields and ParallelIndexSupervisorTask to populate the segments read/published in different parallel compaction algos.

---

Release note

Parallel compaction task completion reports now have segmentsRead and segmentsPublished fields to see how effective a compaction task is.

---

Key changed/added classes in this PR

• ParallelIndexSupervisorTask.java
• IngestionStatsAndErrorsTaskReportData.java

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[YongGang]

LGTM

[suneet-s]

I tried these scenarios:

• index_parallel with no additional subtasks (wikipedia) - the task report has segmentsRead: null and segmentsPublished: null in the report
• index_parallel with more than 1 subtask (kttm1) - the task report has segmentsRead: null and segmentsPublished: null in the sub task and segmentsRead: 0 and semgentsPublished: 1 in the index_parallel task
• Issued a compaction task with no subtasks (kttm1) - the task report has segmentsRead: null and segmentsPublished: null
• issued a compaction task with sub tasks (wikipedia) - no reports for the sub tasks, but the compactTask has segmentsRead: 1 segmentsPublished: 1
• index_kafka on kttm - report has segmentsRead: null segmentsPublished: null

Since this change is meant to be scoped to just compaction tasks, I am -1 in it's current implementation because there are many different task reports than have the fields segmentsRead and segmentsPublished added to it returning null and index_parallel tasks that have a report that says it read 0 segments (while true, that's a little confusing).

If these fields did not show up in the task reports where they are not meant to, I think the change would be good.

[suneet-s]

Looks better! Only other thing I noticed is that index_parallel jobs with no subtasks that used to have task reports before this change no longer show a task report after this change. Can you look into that issue please

[adithyachakilam]

@suneet-s, the missing report you pointed out is actually related to change: #15981 I'll make up a follow up PR to address it.

[adithyachakilam]

Here is the fix: #16042

[kfaraz]

Thanks for adding the new fields, @adithyachakilam . I have left some comments that would be more in line with the existing report class structure.

[kfaraz]

Thanks for the changes, @adithyachakilam ! Requested minor changes, otherwise looks good.

[kfaraz]

LGTM, thanks for addressing the comments.