[FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread #17988

Myracle · 2021-12-02T07:30:42Z

What is the purpose of the change

If there are many hive partitions, creating splits will take much time, for example, up to ten minutes. This mr speeds up the process by multi threads for different partitions.

Brief change log

Add class MRSplitsGetter to create splits by multi thread.
Add class HiveTablePartitionSplits to wrap hive partition and the corresponding inputSplits.
Transfer flinkConf to init MRSplitsGetter's thread num.
Adapt the related calls.

Verifying this change

This change is already covered by existing tests, such as HiveTableSourceITCase.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (not documented)

flinkbot · 2021-12-02T07:34:11Z

CI report:

ccc3e62 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

flinkbot · 2021-12-02T07:34:20Z

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 28b8d68 (Thu Dec 02 07:34:19 UTC 2021)

Warnings:

No documentation files were touched! Remember to keep the Flink docs up to date!
This pull request references an unassigned Jira ticket. According to the code contribution guide, tickets need to be assigned before starting with the implementation work.

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

❓ 1. The [description] looks good.
❓ 2. There is [consensus] that the contribution should go into to Flink.
❓ 3. Needs [attention] from.
❓ 4. The change fits into the overall [architecture].
❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
@flinkbot approve all to approve all aspects
@flinkbot approve-until architecture to approve everything until architecture
@flinkbot attention @username1 [@username2 ..] to require somebody's attention
@flinkbot disapprove architecture to remove an approval you gave earlier

luoyuxia

@Myracle Thanks for contribution. Looks general good to me. Just left some comments.
Also, would you like to share how long it takes to split how many hive partitions with single thread and muti-thread?

...ector-hive/src/main/java/org/apache/flink/connectors/hive/ContinuousHiveSplitEnumerator.java

...tors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/MRSplitsGetter.java

Myracle · 2021-12-07T02:50:36Z

We have 5000 partitions for testing. When the thread is 1, it takes about 341 seconds. When the thread is 3, it takes 122 seconds. More threads means less time to split.

...tors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/MRSplitsGetter.java

…i thread

luoyuxia · 2021-12-15T11:39:02Z

lgtm. @wuchong @JingsongLi @lirui-apache Do you have time to review / merge?

wuchong · 2021-12-15T13:00:28Z

Thanks @luoyuxia for the reviewing. I will have a look and merge it.

wuchong

LGTM.

wuchong · 2021-12-21T06:59:51Z

@flinkbot run azure

wuchong · 2021-12-22T06:53:28Z

@flinkbot run azure

wuchong · 2021-12-22T14:53:53Z

...nectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/HiveOptions.java

@@ -51,4 +51,11 @@
                    .withDescription(
                            "If it is false, using flink native writer to write parquet and orc files; "
                                    + "If it is true, using hadoop mapred record writer to write parquet and orc files.");
+
+    public static final ConfigOption<Integer> TABLE_EXEC_HIVE_PARTITION_SPLIT_THREAD_NUM =
+            key("table.exec.hive.partition-split.thread.num")


Sorry @Myracle, I have one more question. Is this configuration just like Hive's hive.load.dynamic.partitions.thread? I think maybe we can call the configuration table.exec.hive.load-partition-splits.thread-num to be closer to the actual behavior, what do you think?

Besides, could you also add this configuration to the Hive documentation ?

@wuchong Good suggestion! I have modified it. Can you review it again? Thank you.

wuchong · 2022-01-04T04:24:05Z

docs/content.zh/docs/connectors/table/hive/hive_read_write.md

@@ -163,6 +163,12 @@ following parameters in `TableConfig` (note that these parameters affect all sou
        <td>Integer</td>
        <td>Sets max infer parallelism for source operator.</td>
    </tr>
+    <tr>
+        <td><h5>table.exec.hive.load-partition-splits.thread-num</h5></td>


This config option is not belong to Source Parallelism Inference, maybe we need to have a separate section, e.g. Load Partiton Splits

@wuchong Sorry for the wrong place. I have modified it. Please review it again. Thanks.

wuchong · 2022-01-06T03:57:06Z

Thanks for the updating @Myracle

…pache#17988)

rmetzger added the component=Connectors/Hive label Dec 2, 2021

luoyuxia reviewed Dec 6, 2021

View reviewed changes

...ector-hive/src/main/java/org/apache/flink/connectors/hive/ContinuousHiveSplitEnumerator.java Show resolved Hide resolved

luoyuxia reviewed Dec 6, 2021

View reviewed changes

...tors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/MRSplitsGetter.java Outdated Show resolved Hide resolved

luoyuxia reviewed Dec 6, 2021

View reviewed changes

...tors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/MRSplitsGetter.java Show resolved Hide resolved

Myracle force-pushed the FLINK-25010-Speed-up-hive-createMRSplits-by-multi-thread branch 4 times, most recently from 66b45d8 to ed14110 Compare December 10, 2021 09:00

luoyuxia reviewed Dec 10, 2021

View reviewed changes

...tors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/MRSplitsGetter.java Show resolved Hide resolved

Myracle force-pushed the FLINK-25010-Speed-up-hive-createMRSplits-by-multi-thread branch from ed14110 to 140a48b Compare December 13, 2021 02:28

luoyuxia reviewed Dec 13, 2021

View reviewed changes

...tors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/MRSplitsGetter.java Outdated Show resolved Hide resolved

[FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by mult…

f015c1d

…i thread

Myracle force-pushed the FLINK-25010-Speed-up-hive-createMRSplits-by-multi-thread branch from 140a48b to f015c1d Compare December 13, 2021 11:05

wuchong approved these changes Dec 21, 2021

View reviewed changes

wuchong force-pushed the FLINK-25010-Speed-up-hive-createMRSplits-by-multi-thread branch from f015c1d to e7be851 Compare December 21, 2021 04:09

wuchong reviewed Dec 22, 2021

View reviewed changes

Myracle force-pushed the FLINK-25010-Speed-up-hive-createMRSplits-by-multi-thread branch from e7be851 to 19a283c Compare December 25, 2021 03:08

wuchong reviewed Jan 4, 2022

View reviewed changes

refactor config and add doc

ccc3e62

Myracle force-pushed the FLINK-25010-Speed-up-hive-createMRSplits-by-multi-thread branch from 19a283c to ccc3e62 Compare January 6, 2022 03:14

wuchong approved these changes Jan 6, 2022

View reviewed changes

wuchong merged commit 2824c90 into apache:master Jan 6, 2022

niklassemmler pushed a commit to niklassemmler/flink that referenced this pull request Feb 3, 2022

[FLINK-25010][hive] Speed up hive's createMRSplits by multi thread (a…

c0ad546

…pache#17988)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread #17988

[FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread #17988

Myracle commented Dec 2, 2021

flinkbot commented Dec 2, 2021 •

edited

flinkbot commented Dec 2, 2021

luoyuxia left a comment •

edited

Myracle commented Dec 7, 2021

luoyuxia commented Dec 15, 2021

wuchong commented Dec 15, 2021

wuchong left a comment

wuchong commented Dec 21, 2021

wuchong commented Dec 22, 2021

wuchong Dec 22, 2021

Myracle Dec 25, 2021 •

edited

wuchong Jan 4, 2022

Myracle Jan 6, 2022

wuchong commented Jan 6, 2022

[FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread #17988

[FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread #17988

Conversation

Myracle commented Dec 2, 2021

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Dec 2, 2021 • edited

CI report:

flinkbot commented Dec 2, 2021

Automated Checks

Review Progress

luoyuxia left a comment • edited

Choose a reason for hiding this comment

Myracle commented Dec 7, 2021

luoyuxia commented Dec 15, 2021

wuchong commented Dec 15, 2021

wuchong left a comment

Choose a reason for hiding this comment

wuchong commented Dec 21, 2021

wuchong commented Dec 22, 2021

wuchong Dec 22, 2021

Choose a reason for hiding this comment

Myracle Dec 25, 2021 • edited

Choose a reason for hiding this comment

wuchong Jan 4, 2022

Choose a reason for hiding this comment

Myracle Jan 6, 2022

Choose a reason for hiding this comment

wuchong commented Jan 6, 2022

flinkbot commented Dec 2, 2021 •

edited

luoyuxia left a comment •

edited

Myracle Dec 25, 2021 •

edited