Skip to content

[HUDI-5342] Add new bulk insert sort modes repartitioning data by partition path#7402

Merged
alexeykudinkin merged 4 commits intoapache:masterfrom
yihua:HUDI-5342-add-partition-path-redistribute-sort-mode
Dec 10, 2022
Merged

[HUDI-5342] Add new bulk insert sort modes repartitioning data by partition path#7402
alexeykudinkin merged 4 commits intoapache:masterfrom
yihua:HUDI-5342-add-partition-path-redistribute-sort-mode

Conversation

@yihua
Copy link
Contributor

@yihua yihua commented Dec 7, 2022 •
edited
Loading

Change Logs

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

  • For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
  • For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.

New unit tests are added to verify the added functionality.

Impact

This PR adds a new bulk insert sort mode. Existing sort modes are not affected.

Risk level

none

Documentation Update

This PR revises docs for hoodie.bulkinsert.sort.mode in HoodieWriteConfig.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@codope codope added priority:blocker Production down; release blocker writer-core release-0.12.2 Patches targetted for 0.12.2 labels Dec 8, 2022
@codope codope force-pushed the HUDI-5342-add-partition-path-redistribute-sort-mode branch from 3d67bab to 4d1e882 Compare December 8, 2022 10:38
@codope codope mentioned this pull request Dec 8, 2022
Merged
4 tasks
@yihua yihua force-pushed the HUDI-5342-add-partition-path-redistribute-sort-mode branch from 4d1e882 to b67d424 Compare December 9, 2022 22:48
@yihua yihua force-pushed the HUDI-5342-add-partition-path-redistribute-sort-mode branch from b67d424 to 48a4bd1 Compare December 9, 2022 22:51
@yihua yihua changed the title [HUDI-5342] Add a new bulk insert sort mode redistributing data by partition path [HUDI-5342] Add a new bulk insert sort mode repartitioning data by partition path Dec 9, 2022
@yihua yihua changed the title [HUDI-5342] Add a new bulk insert sort mode repartitioning data by partition path [HUDI-5342] Add new bulk insert sort modes repartitioning data by partition path Dec 10, 2022
@hudi-bot
Copy link
Collaborator

hudi-bot commented Dec 10, 2022

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@alexeykudinkin alexeykudinkin merged commit ca3333d into apache:master Dec 10, 2022
nsivabalan pushed a commit that referenced this pull request Dec 13, 2022
@yihua @nsivabalan
[HUDI-5342] Add new bulk insert sort modes repartitioning data by par…
e4bdb73 
…tition path (#7402)

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.
New unit tests are added to verify the added functionality.
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
@yihua
[HUDI-5342] Add new bulk insert sort modes repartitioning data by par…
f9db2e0 
…tition path (apache#7402)

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.
New unit tests are added to verify the added functionality.
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
@yihua
[HUDI-5342] Add new bulk insert sort modes repartitioning data by par…
7af6d29 
…tition path (apache#7402)

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.
New unit tests are added to verify the added functionality.
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
@yihua
[HUDI-5342] Add new bulk insert sort modes repartitioning data by par…
533ceb4 
…tition path (apache#7402)

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.
New unit tests are added to verify the added functionality.
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
@yihua
[HUDI-5342] Add new bulk insert sort modes repartitioning data by par…
73ad817 
…tition path (apache#7402)

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.
New unit tests are added to verify the added functionality.
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
@yihua
[HUDI-5342] Add new bulk insert sort modes repartitioning data by par…
1d47163 
…tition path (apache#7402)

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.
New unit tests are added to verify the added functionality.
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
@yihua
[HUDI-5342] Add new bulk insert sort modes repartitioning data by par…
f33a430 
…tition path (apache#7402)

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.
New unit tests are added to verify the added functionality.
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
@yihua
[HUDI-5342] Add new bulk insert sort modes repartitioning data by par…
bca85f3 
…tition path (apache#7402)

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.
New unit tests are added to verify the added functionality.
alexeykudinkin pushed a commit that referenced this pull request Dec 14, 2022
@yihua
[HUDI-5342] Add new bulk insert sort modes repartitioning data by par…
fb552ae 
…tition path (#7402)

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.
New unit tests are added to verify the added functionality.
fengjian428 pushed a commit to fengjian428/hudi that referenced this pull request Apr 5, 2023
@yihua
[HUDI-5342] Add new bulk insert sort modes repartitioning data by par…
7170f05 
…tition path (apache#7402)

This PR adds two new bulk insert sort modes, PARTITION_PATH_REPARTITION and PARTITION_PATH_REPARTITION_AND_SORT, which does the following

For a physically partitioned table, repartition the input records based on the partition path, limiting the shuffle parallelism to specified outputSparkPartitions. For PARTITION_PATH_REPARTITION_AND_SORT, an additional step of sorting the records based on the partition path within each Spark partition is done.
For a physically non-partitioned table, simply does coalesce for the input rows with outputSparkPartitions.
New unit tests are added to verify the added functionality.
@hudi-bot hudi-bot mentioned this pull request Dec 9, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@codope codope codope left review comments

@alexeykudinkin alexeykudinkin alexeykudinkin approved these changes

Assignees

@alexeykudinkin alexeykudinkin

@codope codope

Labels

priority:blocker Production down; release blocker release-0.12.2 Patches targetted for 0.12.2

Projects

Hudi PR Support
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yihua @hudi-bot @alexeykudinkin @codope