[HUDI-6481] Support run multi tables services in a single spark job #9558

stream2000 · 2023-08-28T11:12:48Z

Change Logs

Origin Issue: #8960

Now we have HoodieMultiTableDeltaStreamer using spark to ingest multi tables into hudi, and we can also ingest multi tables using a single flink in some platforms like alicloud vvp. In such scenario we may want to run all tables services outside of the ingestion job to improve stability.

Currently, hudi provides a bunch of offline table service jobs but we still need to run them manually or through some schedule tools. This Pr provides a tool to run all table services in a single spark job

Impact

provides a tool to run all table services in a single spark job

Risk level (write none, low medium or high below)

none

Documentation Update

Will update document after merge

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

stream2000 · 2023-08-29T01:46:39Z

@leesf @jonvex Hi, could you please help review this pr? Thanks~

jonvex

Good job! Code looks solid. @nsivabalan @xushiyan @danny0405 @leesf can one of you take a look at HoodieMultiTableServicesMain and validate that this aligns with how we want table services to be run? Also the use of concurrency instead of running sequentially?

jonvex · 2023-08-29T15:17:23Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java

+    this.jsc = jsc;
+    this.props = props;
+    this.metaClient = UtilHelpers.createMetaClient(jsc, cfg.basePath, true);
+  }


Do we want to do the disable async cleaning and add lock options like the other constructor?

Sure! Thanks for reminding

jonvex · 2023-08-29T15:17:53Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java

+    this.jsc = jsc;
+    this.props = props;
+    this.metaClient = UtilHelpers.createMetaClient(jsc, cfg.basePath, true);
+  }


Same question as in clustering job

sure and updated

jonvex · 2023-08-29T15:18:40Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java

+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class ArchiveTask extends TableServiceTask {


add javadocs

Added javadoc for all new classes

jonvex · 2023-08-29T15:19:53Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/CleanTask.java

+
+import org.apache.spark.api.java.JavaSparkContext;
+
+class CleanTask extends TableServiceTask {


jonvex · 2023-08-29T15:20:25Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ClusteringTask.java

+
+import org.apache.spark.api.java.JavaSparkContext;
+
+class ClusteringTask extends TableServiceTask {


jonvex · 2023-08-29T15:36:02Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/TableServicePipeline.java

+import java.util.ArrayList;
+import java.util.List;
+
+public class TableServicePipeline {


jonvex · 2023-08-29T15:37:18Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java

+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *    http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */


In all the files you have double commented, it should just be

/* * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, * software distributed under the License is distributed on an * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * KIND, either express or implied. See the License for the * specific language governing permissions and limitations * under the License. */

updated all license header

jonvex · 2023-08-29T15:40:42Z

...ies/src/test/java/org/apache/hudi/utilities/multitable/TestHoodieMultiTableServicesMain.java

+        LOG.info("Shutdown the table services");
+        main.cancel();
+      } catch (InterruptedException e) {
+        //


updated and ignore the exception

jonvex · 2023-08-29T15:42:12Z

...ies/src/test/java/org/apache/hudi/utilities/multitable/TestHoodieMultiTableServicesMain.java

+        .withParallelism(4, 4)
+        .withBulkInsertParallelism(4)
+        .withFinalizeWriteParallelism(2)
+        .withProps(makeIndexConfig(HoodieIndex.IndexType.BUCKET))


why bucket index? Just curious

Just want to make it easier to produce log for compaciton.

jonvex · 2023-08-29T15:57:06Z

...ies/src/test/java/org/apache/hudi/utilities/multitable/TestHoodieMultiTableServicesMain.java

+
+import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;
+
+class TestHoodieMultiTableServicesMain extends HoodieCommonTestHarness implements SparkProvider {


jonvex · 2023-08-29T16:06:32Z

@stream2000 also please sign up for jira and email the dev mailing list if you haven't. And assign yourself to the jira ticket.

stream2000

Thanks for your review! ~ I have addressed all comments and assigned the jira to myself

stream2000 · 2023-08-31T07:10:31Z

...ilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java

+  private ScheduledExecutorService executorService;
+
+  private void batchRunTableServices(List<String> tablePaths) throws InterruptedException, ExecutionException {
+    ExecutorService executorService = Executors.newFixedThreadPool(cfg.poolSize);


I don't really understand what you mean here? I created a new local thread pool for running all table services and destroyed it since we only need it once.

leesf · 2023-08-31T09:44:26Z

We have already used this feature internally to make table service management of Hudi tables easier.

leesf · 2023-08-31T13:35:30Z

...ilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java

+ * Main function for executing multi-table services
+ */
+public class HoodieMultiTableServicesMain {
+  private static final Logger LOG = LoggerFactory.getLogger(HoodieStreamer.class);


HoodieMultiTableServicesMain.class?

Sure, I missed it when migrated it from HoodieStreamer. Will check other classes also.

leesf · 2023-08-31T13:39:10Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/MultiTableServiceUtils.java

+
+      if (!dirs.isEmpty()) {
+        List<Pair<FileStatus, Integer>> dirResults = engineContext.map(dirs, fileStatus -> {
+          if (isHoodieTable(fileStatus.getPath(), conf.get())) {


what does 0, 1, 2 mean? looks like magic number

Nice catch~ It's not a good design that uses hard-coded magic number, already updated the magic number to meaningful enum constants.

Updated to:

/** * Type of directories when searching hoodie tables under path */ enum DirType { HOODIE_TABLE, // previous 0 NORMAL_DIR, // previous 1 META_FOLDER // previous 2 }

leesf · 2023-09-01T00:54:55Z

...ilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java

+    ExecutorService executorService = Executors.newFixedThreadPool(cfg.poolSize);
+    List<CompletableFuture<Void>> futures = tablePaths.stream()
+        .map(basePath -> CompletableFuture.runAsync(
+            () -> MultiTableServiceUtils.buildTableServicePipeline(jsc, basePath, cfg, props).execute(),


should early exit if no services is enabled?

If no services is enabled, the TableServicePipeline will contain no task and the execute method will return directly. So it's OK to keep current design.

stream2000

Thanks for your review! Already addressed all comments, PTAL

stream2000 · 2023-09-01T03:00:44Z

...ilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java

+    ExecutorService executorService = Executors.newFixedThreadPool(cfg.poolSize);
+    List<CompletableFuture<Void>> futures = tablePaths.stream()
+        .map(basePath -> CompletableFuture.runAsync(
+            () -> MultiTableServiceUtils.buildTableServicePipeline(jsc, basePath, cfg, props).execute(),


If no services is enabled, the TableServicePipeline will contain no task and the execute method will return directly. So it's OK to keep current design.

stream2000 · 2023-09-01T03:01:29Z

...ilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java

+ * Main function for executing multi-table services
+ */
+public class HoodieMultiTableServicesMain {
+  private static final Logger LOG = LoggerFactory.getLogger(HoodieStreamer.class);


Sure, I missed it when migrated it from HoodieStreamer. Will check other classes also.

stream2000 · 2023-09-01T03:17:09Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/MultiTableServiceUtils.java

+
+      if (!dirs.isEmpty()) {
+        List<Pair<FileStatus, Integer>> dirResults = engineContext.map(dirs, fileStatus -> {
+          if (isHoodieTable(fileStatus.getPath(), conf.get())) {


Nice catch~ It's not a good design that uses hard-coded magic number, already updated the magic number to meaningful enum constants.

Updated to:

/** * Type of directories when searching hoodie tables under path */ enum DirType { HOODIE_TABLE, // previous 0 NORMAL_DIR, // previous 1 META_FOLDER // previous 2 }

stream2000 · 2023-09-02T00:18:38Z

@hudi-bot run azure

stream2000 · 2023-09-04T02:13:57Z

@leesf @jonvex Hi, thanks for your review! I have addressed all comments and can we move on this~

leesf

LGTM, @jonvex any other comments?

danny0405 · 2023-09-05T03:01:46Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java

+  public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+    this.cfg = cfg;
+    this.jssc = jssc;
+    this.props = props;


The original code fetches the props either from cmd line or global options, what is the purpose to support custom props per-service then?

Yes in multi-table services we also fetch the props either from cmd line or global options. And we do not want each service to fetch the props by themselves. So here I add a new constructor to use the props that all services shared.

n multi-table services we also fetch the props either from cmd line or global options

The existing constructor already satisfy the needs.

The existing constructor already satisfy the needs.

Yes, however if we want to reuse existing constructor, we need pass the prop file path and the cmd line config from HoodieMultiTableServicesMain to HoodieCleaner. And we will parse the config(read props file and use cmd line config to override it) for every TableServiceTask , which is redundant. Current implementation will do the config parsing in HoodieMultiTableServicesMain only, and all the TableServiceTask will use the final config parsed by HoodieMultiTableServicesMain. What do you think?

Fine with it.

Then we can force one constructor to invoke the other one.

Then we can make one constructor to invokes the other one.

Thanks for your suggestions~ I have slightly refactored the code and make it cleaner, PTAL

danny0405 · 2023-09-05T03:03:06Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java

+import org.slf4j.LoggerFactory;
+
+/**
+ * Archive task to run in TableServicePipeline


Every doc sentence should end up with dot .

danny0405 · 2023-09-05T03:03:47Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java

+  public static Builder newBuilder() {
+    return new Builder();
+  }
+


Add doc for each Builder class, also the doc for each member variables.

stream2000

@danny0405, Thanks for your review! All comments addressed and added a new param compactionStrategyClassName to config since I missed it in previous version.

stream2000 · 2023-09-05T08:22:43Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java

+import org.slf4j.LoggerFactory;
+
+/**
+ * Archive task to run in TableServicePipeline


stream2000 · 2023-09-05T08:23:06Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java

+  public static Builder newBuilder() {
+    return new Builder();
+  }
+


jonvex

Hi. Sorry I was out of the country. Looks good

stream2000 · 2023-09-06T05:39:56Z

@hudi-bot run azure

stream2000 · 2023-09-06T10:18:37Z

@hudi-bot run azure

hudi-bot · 2023-09-07T10:37:36Z

CI report:

b67857b Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

leesf · 2023-09-07T11:07:09Z

Merging this as CI failed with other reason.

…pache#9558)

jonvex requested changes Aug 29, 2023

View reviewed changes

stream2000 commented Aug 31, 2023

View reviewed changes

stream2000 force-pushed the HUDI-6481_multi_table_services branch from bb592b7 to d0a5621 Compare August 31, 2023 07:22

leesf reviewed Aug 31, 2023

View reviewed changes

leesf reviewed Sep 1, 2023

View reviewed changes

stream2000 commented Sep 1, 2023

View reviewed changes

stream2000 force-pushed the HUDI-6481_multi_table_services branch from ffc0272 to 84d10c9 Compare September 2, 2023 13:40

leesf approved these changes Sep 4, 2023

View reviewed changes

danny0405 reviewed Sep 5, 2023

View reviewed changes

stream2000 commented Sep 5, 2023

View reviewed changes

stream2000 force-pushed the HUDI-6481_multi_table_services branch from aa20e8b to e7a6ae7 Compare September 5, 2023 09:21

jonvex approved these changes Sep 5, 2023

View reviewed changes

stream2000 added 5 commits September 6, 2023 23:02

[HUDI-6481] Support run multi tables services in a single spark job

9dc241c

addressed comments

df92b3d

addressed comments

ce690af

fix checktyle

974126f

improve javadoc & add compactionStrategyClassName to config

31dabb9

stream2000 force-pushed the HUDI-6481_multi_table_services branch from e7a6ae7 to 31dabb9 Compare September 6, 2023 15:27

addressed comments

b67857b

danny0405 approved these changes Sep 7, 2023

View reviewed changes

leesf merged commit bbeb531 into apache:master Sep 7, 2023
27 checks passed

leosanqing pushed a commit to leosanqing/hudi that referenced this pull request Sep 13, 2023

[HUDI-6481] Support run multi tables services in a single spark job (a…

3c47064

…pache#9558)

vinishjail97 pushed a commit to vinishjail97/hudi that referenced this pull request Feb 16, 2024

[HUDI-6481] Support run multi tables services in a single spark job (a…

87e9664

…pache#9558)


		import org.apache.spark.api.java.JavaSparkContext;

		class CleanTask extends TableServiceTask {


		import org.apache.spark.api.java.JavaSparkContext;

		class ClusteringTask extends TableServiceTask {


		import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;

		class TestHoodieMultiTableServicesMain extends HoodieCommonTestHarness implements SparkProvider {

[HUDI-6481] Support run multi tables services in a single spark job #9558

[HUDI-6481] Support run multi tables services in a single spark job #9558

Conversation

stream2000 commented Aug 28, 2023 • edited

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

stream2000 commented Aug 29, 2023

jonvex left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonvex commented Aug 29, 2023

stream2000 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leesf commented Aug 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leesf Aug 31, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stream2000 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stream2000 commented Sep 2, 2023

stream2000 commented Sep 4, 2023

leesf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stream2000 Sep 6, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danny0405 Sep 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stream2000 left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonvex left a comment

Choose a reason for hiding this comment

stream2000 commented Sep 6, 2023

stream2000 commented Sep 6, 2023

hudi-bot commented Sep 7, 2023

CI report:

leesf commented Sep 7, 2023

stream2000 commented Aug 28, 2023 •

edited

jonvex left a comment •

edited

leesf Aug 31, 2023 •

edited

stream2000 Sep 6, 2023 •

edited

danny0405 Sep 7, 2023 •

edited

stream2000 left a comment •

edited