Cleanup serialiazation of TaskReportMap #16217

kfaraz · 2024-03-29T07:57:20Z

Issue

While serializing a Map or even a List containing TaskReport objects, the type information is lost. Thus, the object cannot be serialized back.

This is a known issue with Jackson.

Existing solution in Druid

The serialization of a Map<String, TaskReport> was originally fixed in #12938.

The way this has been tackled in the Druid code till now is:

Use custom serialisation logic in SingleFileTaskReportFileWriter.writeReportToStream() to write out each TaskReport object one by one
For live reports that are served over HTTP, build a raw map instead of a concrete TaskReport object

Proposed solution

Add a new ReportMap class.

Changes

Add class TaskReport.ReportMap
Have TaskReport.buildTaskReports() return the new class
Add serde tests for writing to string, file for the known task report types
Replace all occurrences of Map<String, TaskReport> with TaskReport.ReportMap to ensure
that we always use this class for serialization of reports
Add method AbstractBatchIndexTask.buildLiveIngestionStatsReport() to reduce duplication and hard-coding of serializable field names.

Important classes

TaskReport
AbstractBatchIndexTask
SingleFileTaskReportFileWriter
ParallelIndexSupervisorTask
SinglePhaseSubTask
IndexTask

Rolling upgrade concerns

None

Serialization: results in the same json as before
Deserialization: works with the same json as before

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

…nto cleanup_build_task_report

…cleanup

abhishekrb19 · 2024-04-01T02:54:49Z

indexing-service/src/test/java/org/apache/druid/indexing/common/task/TaskReportSerdeTest.java

    Assert.assertEquals(reportMap1, reportMap2);
  }

+  @Test
+  public void testWriteReportMapToStringAndRead() throws Exception


Could we also add a test to verify that a serialized old type task report Map<String, TaskReport> deserializes correctly into the new type TaskReport.ReportMap? Should address any upgrade concerns.

Ditto for the reverse roundtrip for a downgrade scenario.

Sure, that makes sense.

abhishekrb19 · 2024-04-01T03:00:48Z

indexing-service/src/main/java/org/apache/druid/indexing/common/TaskReport.java

+   * a TaskReport is serialized without the type information and cannot be
+   * deserialized back into a concrete implementation.
+   */
+  class ReportMap extends LinkedHashMap<String, TaskReport>


Are there any tests that verify the reports are indeed ordered since we rely on a LinkedHashMap? Just looking at the callers of buildTaskReports(), I don't seem to find any.

No, I can add a test to verify the order. Although, I don't see any actual task writing a report map that contains multiple entries. Also not sure why the order was considered to be important in the first place, its json anyway.

abhishekrb19 · 2024-04-01T03:04:33Z

...pache/druid/indexing/common/task/batch/parallel/AbstractParallelIndexSupervisorTaskTest.java

@@ -546,12 +548,17 @@ public ListenableFuture<Void> runTask(String taskId, Object taskObject)

    @Override
    public ListenableFuture<Map<String, Object>> taskReportAsMap(String taskId)
+    {
+      return Futures.immediateFuture(null);


Should this call getLiveReportsForTask(taskId)?

It was doing that originally but not needed right now as I have added the other method getLiveReportsForTask() just below this one.

This is anyway used only in the tests and I plan to fix it back up once I replace the Map<String, Object> in the OverlordClient with TaskReport.ReportMap.

Ok, makes sense

abhishekrb19 · 2024-04-01T03:04:48Z

...pache/druid/indexing/common/task/batch/parallel/AbstractParallelIndexSupervisorTaskTest.java

+      return Futures.immediateFuture(null);
+    }
+
+    public TaskReport.ReportMap getLiveReportsForTask(String taskId)


Suggested change

public TaskReport.ReportMap getLiveReportsForTask(String taskId)

protected TaskReport.ReportMap getLiveReportsForTask(String taskId)

abhishekrb19 · 2024-04-01T03:09:44Z

...pache/druid/indexing/common/task/batch/parallel/AbstractParallelIndexSupervisorTaskTest.java

@@ -546,12 +548,17 @@ public ListenableFuture<Void> runTask(String taskId, Object taskObject)

    @Override
    public ListenableFuture<Map<String, Object>> taskReportAsMap(String taskId)


Can taskReportAsMap() now return the concrete type TaskReport.ReportMap instead of Map<String, Object>?

Yes, I have that change in a follow up PR. Didn't do it here as it requires moving all the TaskReport related classes to the druid-processing module, so that OverlordClient can use it.

kfaraz · 2024-04-01T03:34:20Z

Thanks a lot for the review, @abhishekrb19 !
Do you think I can include your suggestions in my follow up PR as they are all related to tests?
I don't want to invoke another CI cycle if I can help it 😛 .

abhishekrb19

LGTM, thanks! I'm ok with doing the suggestions in a follow-up 👍

Follow up to #16217 Changes: - Update `OverlordClient.getReportAsMap()` to return `TaskReport.ReportMap` - Move the following classes to `org.apache.druid.indexer.report` in the `druid-processing` module - `TaskReport` - `KillTaskReport` - `IngestionStatsAndErrorsTaskReport` - `TaskContextReport` - `TaskReportFileWriter` - `SingleFileTaskReportFileWriter` - `TaskReportSerdeTest` - Remove `MsqOverlordResourceTestClient` as it had only one method which is already present in `OverlordResourceTestClient` itself

kfaraz and others added 8 commits March 26, 2024 14:56

Build task reports in AbstractBatchIndexTask

eb5b833

Minor cleanup

b5bb6e6

Apply suggestions from code review by @abhishekrb

a82d20b

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

Cleanup IndexTaskTest

8c96995

Merge branch 'cleanup_build_task_report' of github.com:kfaraz/druid i…

8f6f2ba

…nto cleanup_build_task_report

Fix formatting

b133e77

Fix coverage

32b8442

Cleanup serialization of TaskReport map

6a8bbe7

github-actions bot added Area - Streaming Ingestion Area - Ingestion labels Mar 29, 2024

Merge branch 'master' of github.com:apache/druid into further_report_…

2ca1554

…cleanup

kfaraz requested review from abhishekrb19 and gianm March 29, 2024 08:00

Replace occurrences of Map<String, TaskReport>

ae943b7

github-actions bot added Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Mar 29, 2024

kfaraz added 3 commits March 30, 2024 07:43

Merge branch 'master' of github.com:apache/druid into further_report_…

0be5a1e

…cleanup

Return TaskReport.ReportMap for live reports, fix test comparisons

172431a

Address test failures

d25ff7e

abhishekrb19 reviewed Apr 1, 2024

View reviewed changes

abhishekrb19 approved these changes Apr 1, 2024

View reviewed changes

gianm approved these changes Apr 1, 2024

View reviewed changes

gianm merged commit 0de44d9 into apache:master Apr 1, 2024
85 checks passed

kfaraz deleted the further_report_cleanup branch April 1, 2024 19:23

kfaraz mentioned this pull request Apr 2, 2024

Fix OverlordClient to read reports as a concrete ReportMap object instead of a generic map #16226

Merged

10 tasks

adarshsanjeev added this to the 30.0.0 milestone May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cleanup serialiazation of TaskReportMap #16217

Cleanup serialiazation of TaskReportMap #16217

kfaraz commented Mar 29, 2024 •

edited

abhishekrb19 Apr 1, 2024

kfaraz Apr 1, 2024

abhishekrb19 Apr 1, 2024

kfaraz Apr 1, 2024 •

edited

abhishekrb19 Apr 1, 2024

kfaraz Apr 1, 2024

abhishekrb19 Apr 1, 2024

abhishekrb19 Apr 1, 2024

abhishekrb19 Apr 1, 2024

kfaraz Apr 1, 2024

kfaraz commented Apr 1, 2024

abhishekrb19 left a comment

	public TaskReport.ReportMap getLiveReportsForTask(String taskId)
	protected TaskReport.ReportMap getLiveReportsForTask(String taskId)

		@@ -546,12 +548,17 @@ public ListenableFuture<Void> runTask(String taskId, Object taskObject)

		@Override
		public ListenableFuture<Map<String, Object>> taskReportAsMap(String taskId)

Cleanup serialiazation of TaskReportMap #16217

Cleanup serialiazation of TaskReportMap #16217

Conversation

kfaraz commented Mar 29, 2024 • edited

Issue

Existing solution in Druid

Proposed solution

Changes

Important classes

Rolling upgrade concerns

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kfaraz Apr 1, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kfaraz commented Apr 1, 2024

abhishekrb19 left a comment

Choose a reason for hiding this comment

kfaraz commented Mar 29, 2024 •

edited

kfaraz Apr 1, 2024 •

edited