Auto compaction based on parallel indexing #8570

jihoonson · 2019-09-23T02:57:55Z

Description

This PR is to allow for auto compaction to use the parallel indexing task. This will be useful when there are too many or large segments in a single time chunk.

New/changed configurations

Parallel indexing task has a new configuration, splitHintSpec, in the tuningConfig to allow for operators to give a hint to control the amount of data that each first phase sub task reads. SegmentsSplitHintSpec is the only available option for now which is used only for IngestSegmentFirehose.
Compaction task now uses ParallelIndexTuningConfig.
Auto compaction tuning config now supports maxNumConcurrentSubTasks and splitHintSpec.

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added unit tests or modified existing tests to cover new code paths.
been tested in a test Druid cluster.

This change is

ccaominh · 2019-10-02T17:41:18Z

core/src/main/java/org/apache/druid/data/input/SegmentsSplitHintSpec.java

+public class SegmentsSplitHintSpec implements SplitHintSpec
+{
+  public static final String TYPE = "segments";
+  public static final long DEFAULT_MAX_INPUT_SEGMENT_BYTES_PER_TASK = 150 * 1024 * 1024;


Access can be private

Thanks, fixed.

ccaominh · 2019-10-02T17:42:53Z

core/src/main/java/org/apache/druid/data/input/SegmentsSplitHintSpec.java

+      @JsonProperty("maxInputSegmentBytesPerTask") @Nullable Long maxInputSegmentBytesPerTask
+  )
+  {
+    this.maxInputSegmentBytesPerTask = maxInputSegmentBytesPerTask == null


Do we have to handle -1 as null for new specs?

This parameter was added in #7048 and it doesn't count -1 as null. IMO, handling -1 as null is a legacy behavior and new parameters shouldn't do that.

ccaominh · 2019-10-02T17:44:02Z

docs/configuration/index.md

@@ -815,9 +815,11 @@ If you see this problem, it's recommended to set `skipOffsetFromLatest` to some
 |`maxRowsInMemory`|See [tuningConfig for indexTask](../ingestion/native-batch.md#tuningconfig)|no (default = 1000000)|
 |`maxBytesInMemory`|See [tuningConfig for indexTask](../ingestion/native-batch.md#tuningconfig)|no (1/6 of max JVM memory)|
 |`maxTotalRows`|See [tuningConfig for indexTask](../ingestion/native-batch.md#tuningconfig)|no (default = 20000000)|
+|`splitHintSpec`|See [tuningConfig for indexTask](../ingestion/native-batch.md#tuningconfig)|no (default = null|


typo: null -> null)

Fixed, thanks.

ccaominh · 2019-10-02T17:57:47Z

...ain/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexTuningConfig.java

+    return new ParallelIndexTuningConfig(
+        null,
+        null,
+        getMaxRowsInMemory(),
+        getMaxBytesInMemory(),
+        null,
+        null,
+        splitHintSpec,
+        partitionsSpec,
+        getIndexSpec(),
+        getIndexSpecForIntermediatePersists(),
+        getMaxPendingPersists(),
+        isForceGuaranteedRollup(),
+        isReportParseExceptions(),
+        getPushTimeout(),
+        getSegmentWriteOutMediumFactory(),
+        null,
+        maxNumConcurrentSubTasks,
+        maxRetry,
+        taskStatusCheckPeriodMs,
+        chatHandlerTimeout,
+        chatHandlerNumRetries,
+        maxNumSegmentsToMerge,
+        totalNumMergeTasks,
+        isLogParseExceptions(),
+        getMaxParseExceptions(),
+        getMaxSavedParseExceptions()
+    );


Possibly make consistent to use either getters for all parameters or direct access for all

Changed to use getters.

ccaominh · 2019-10-02T18:24:32Z

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java

+    );
+    final List<ParallelIndexSupervisorTask> indexTaskSpecs = IntStream
+        .range(0, ingestionSpecs.size())
+        .mapToObj(i -> {


Cool! Didn't know about IntStream.

ccaominh · 2019-10-02T18:56:55Z

...c/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorSegmentCompactor.java

+   * compaction tasks, we should count the sub tasks of parallel indexing task as well. However, we currently
+   * don't have a way to get the number of current running sub tasks except poking each supervisor task,
+   * which is complex to handle all kinds of failures.
+   * Here, instead, we compute a rough number of running sub tasks by summing maxNumConcurrentSubTasks


Comment is misleading as the summation is done in the caller of this method. Perhaps the comment should be moved/reworded.

Oops, forgot to update it after I moved this javadoc. Fixed.

ccaominh · 2019-10-02T19:03:36Z

...c/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorSegmentCompactor.java

+    if (tuningConfig != null && tuningConfig.getMaxNumConcurrentSubTasks() != null) {
+      // The actual number of subtasks might be smaller than the configured max.
+      // However, we use the max to simplify the estimation here.
+      return tuningConfig.getMaxNumConcurrentSubTasks();
+    } else {
+      return 0;
+    }


tuningConfig defaults to a value of 1 for maxNumConcurrentSubTasks. Is that inconsistent with this method returning a value of 0 if tuningConfig/maxNumConcurrentSubTasks is missing?

If maxNumConcurrentSubTasks is 1, the supervisor task runs in the sequential mode and processes data by itself instead of spawning sub tasks.

ccaominh · 2019-10-02T19:08:57Z

...c/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorSegmentCompactor.java

@@ -193,6 +222,7 @@ private CoordinatorStats doRun(
            taskId,
            Iterables.transform(segmentsToCompact, DataSegment::getId)
        );
+        numSubmittedTasks += findNumMaxConcurrentSubTasks(config.getTuningConfig()) + 1;


Comment for the + 1 (similar to what you have for line 116) may be useful

ccaominh · 2019-10-02T20:03:58Z

...rvice/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskParallelRunTest.java

+
+  private static ParallelIndexTuningConfig newTuningConfig()
+  {
+    return new ParallelIndexTuningConfig(


It's unfortunate that there isn't a builder

ccaominh · 2019-10-02T20:25:34Z

server/src/main/java/org/apache/druid/client/indexing/ClientCompactQueryTuningConfig.java

@@ -54,11 +59,15 @@ public static ClientCompactQueryTuningConfig from(
        userCompactionTaskQueryTuningConfig == null ? null : userCompactionTaskQueryTuningConfig.getMaxRowsInMemory(),
        userCompactionTaskQueryTuningConfig == null ? null : userCompactionTaskQueryTuningConfig.getMaxBytesInMemory(),
        userCompactionTaskQueryTuningConfig == null ? null : userCompactionTaskQueryTuningConfig.getMaxTotalRows(),
+        userCompactionTaskQueryTuningConfig == null ? null : userCompactionTaskQueryTuningConfig.getSplitHintSpec(),


May be more readable if the check for null userCompactionTaskQueryTuningConfig is moved up:

if (userCompactionTaskQueryTuningConfig == null) { return new ClientCompactQueryTuningConfig( maxRowsPerSegment, null, null, ... ) } else { return new ClientCompactQueryTuningConfig( maxRowsPerSegment, userCompactionTaskQueryTuningConfig.getMaxRowsInMemory(), userCompactionTaskQueryTuningConfig.getMaxBytesInMemory(), ... ) }

Good point. Fixed.

…lel-compact-master

ccaominh · 2019-10-09T17:15:25Z

...n/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java

          LOG.warn(
-              "maxNumConcurrentSubTasks is 1. Running sequentially. "
-              + "Please set maxNumConcurrentSubTasks to something higher than 1 if you want to run in parallel ingestion mode."
+              "maxNumConcurrentSubTasks[%s] is less than 1. Running sequentially. Please set maxNumConcurrentSubTasks "


Message should say "less than or equal to 1"?

Thanks! Fixed.

…lel-compact-master

ccaominh

LGTM 👍

clintropolis

+1 after resolve conflicts (and CI)

clintropolis · 2019-10-17T23:58:29Z

core/src/main/java/org/apache/druid/data/input/SegmentsSplitHintSpec.java

+{
+  public static final String TYPE = "segments";
+
+  private static final long DEFAULT_MAX_INPUT_SEGMENT_BYTES_PER_TASK = 150 * 1024 * 1024;


Should this value be larger maybe?

Thanks, increased to 500MB.

…lel-compact-master

jihoonson · 2019-10-18T20:24:09Z

@ccaominh @clintropolis thank you for the review!

glasser · 2019-11-15T23:32:02Z

Does this still create a compact task that uses parallel indexing internally, or does it actually generate an index_parallel task?

jihoonson added 3 commits September 23, 2019 10:41

Auto compaction based on parallel indexing

8cdbc61

javadoc and doc

f77dfb7

typo

9a60024

jihoonson added Area - Batch Ingestion Release Notes labels Sep 23, 2019

update spell

2e0b7a8

ccaominh reviewed Oct 2, 2019

View reviewed changes

jihoonson added 3 commits October 4, 2019 15:02

addressing comments

d7ceba6

Merge branch 'master' of github.com:apache/incubator-druid into paral…

459b0b0

…lel-compact-master

address comments

abce246

ccaominh reviewed Oct 9, 2019

View reviewed changes

jihoonson added 4 commits October 9, 2019 17:31

Merge branch 'master' of github.com:apache/incubator-druid into paral…

ff81606

…lel-compact-master

fix log

a4f1a66

fix build

cc50489

fix test

fe131de

ccaominh approved these changes Oct 10, 2019

View reviewed changes

jihoonson mentioned this pull request Oct 14, 2019

Stateful auto compaction #8573

Merged

5 tasks

clintropolis reviewed Oct 17, 2019

View reviewed changes

clintropolis approved these changes Oct 17, 2019

View reviewed changes

jihoonson added 3 commits October 17, 2019 17:43

Merge branch 'master' of github.com:apache/incubator-druid into paral…

06d5c36

…lel-compact-master

increase default max input segment bytes per task

dc3bcab

fix test

f68c3a4

jihoonson merged commit 30c1590 into apache:master Oct 18, 2019

leventov mentioned this pull request Nov 14, 2019

splitHintSpec is not added to ClientCompactQueryTuningConfig's equals(), hashCode(), and toString() #8866

Closed

jon-wei added this to the 0.17.0 milestone Dec 17, 2019

jon-wei mentioned this pull request Dec 28, 2019

0.17.0 release notes #9066

Closed

jihoonson mentioned this pull request Jan 18, 2020

Support both IndexTuningConfig and ParallelIndexTuningConfig for compaction task #9222

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto compaction based on parallel indexing #8570

Auto compaction based on parallel indexing #8570

jihoonson commented Sep 23, 2019 •

edited

Loading

ccaominh Oct 2, 2019

jihoonson Oct 9, 2019

ccaominh Oct 2, 2019

jihoonson Oct 9, 2019

ccaominh Oct 2, 2019

jihoonson Oct 9, 2019

ccaominh Oct 2, 2019

jihoonson Oct 9, 2019

ccaominh Oct 2, 2019

ccaominh Oct 2, 2019

jihoonson Oct 9, 2019

ccaominh Oct 2, 2019

jihoonson Oct 9, 2019

ccaominh Oct 2, 2019

jihoonson Oct 9, 2019

ccaominh Oct 2, 2019

jihoonson Oct 9, 2019

ccaominh Oct 2, 2019

jihoonson Oct 9, 2019

ccaominh Oct 9, 2019 •

edited

Loading

jihoonson Oct 10, 2019

ccaominh left a comment

clintropolis left a comment

clintropolis Oct 17, 2019

jihoonson Oct 18, 2019

jihoonson commented Oct 18, 2019

glasser commented Nov 15, 2019

Auto compaction based on parallel indexing #8570

Auto compaction based on parallel indexing #8570

Conversation

jihoonson commented Sep 23, 2019 • edited Loading

Description

New/changed configurations

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ccaominh Oct 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ccaominh left a comment

Choose a reason for hiding this comment

clintropolis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jihoonson commented Oct 18, 2019

glasser commented Nov 15, 2019

jihoonson commented Sep 23, 2019 •

edited

Loading

ccaominh Oct 9, 2019 •

edited

Loading