[HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem #3209

xiarixiaoyao · 2021-07-01T11:39:36Z

…the oom of hive MetaStore

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

when we try to sync 10w partitions to hive by using HiveSyncTool lead to the oom of hive MetaStore。

here is a stress test for HiveSyncTool

env：

hive metastore -Xms16G -Xmx16G

hive.metastore.client.socket.timeout=10800

partitionNum	time consume
100	37s
1000	168s
5000	1830s
10000	timeout
100000	hive metastore oom

HiveSyncTools sync all partitions to hive metastore at once。 when the partitions num is large ，it puts a lot of pressure on hive metastore。 for large partition num we should support batch sync 。

## Brief change log

(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request

UT added

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

hudi-bot · 2021-07-01T11:41:31Z

CI report:

56f820d Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run travis re-run the last Travis build
@hudi-bot run azure re-run the last Azure build

codecov-commenter · 2021-07-01T17:41:49Z

Codecov Report

Merging #3209 (56f820d) into master (62a1ad8) will increase coverage by 0.00%.
The diff coverage is 86.36%.

@@            Coverage Diff            @@
##             master    #3209   +/-   ##
=========================================
  Coverage     47.51%   47.52%           
- Complexity     5431     5435    +4     
=========================================
  Files           922      922           
  Lines         41001    41016   +15     
  Branches       4104     4106    +2     
=========================================
+ Hits          19480    19491   +11     
- Misses        19799    19801    +2     
- Partials       1722     1724    +2

Flag	Coverage Δ
hudicli	`39.97% <ø> (ø)`
hudiclient	`34.57% <ø> (ø)`
hudicommon	`48.38% <ø> (-0.02%)`	⬇️
hudiflink	`59.91% <ø> (ø)`
hudihadoopmr	`51.29% <ø> (ø)`
hudisparkdatasource	`67.26% <100.00%> (+<0.01%)`	⬆️
hudisync	`54.48% <85.71%> (+0.23%)`	⬆️
huditimelineservice	`64.07% <ø> (ø)`
hudiutilities	`58.05% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...in/java/org/apache/hudi/hive/HoodieHiveClient.java	`70.13% <82.35%> (-0.01%)`	⬇️
...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala	`71.18% <100.00%> (+0.08%)`	⬆️
...main/java/org/apache/hudi/hive/HiveSyncConfig.java	`98.07% <100.00%> (+0.11%)`	⬆️
...he/hudi/hive/replication/GlobalHiveSyncConfig.java	`95.00% <100.00%> (+0.26%)`	⬆️
...e/hudi/common/table/log/HoodieLogFormatWriter.java	`78.12% <0.00%> (-1.57%)`	⬇️
...apache/hudi/utilities/deltastreamer/DeltaSync.java	`71.38% <0.00%> (+0.33%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62a1ad8...56f820d. Read the comment docs.

yanghua · 2021-07-02T01:17:28Z

@xiarixiaoyao This title of PR describes the problem, while the better title is to describe "what do you want to do in the PR". WDYT?

xiarixiaoyao · 2021-07-02T04:01:41Z

@yanghua thanks for your review. already changed the PR title.

leesf · 2021-07-02T15:16:28Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala

@@ -471,6 +471,11 @@ object DataSourceWriteOptions {
    .key("hoodie.deltastreamer.source.kafka.value.deserializer.schema")
    .noDefaultValue()
    .withDocumentation("")
+
+  val HIVE_BATCH_SYNC_PARTITION_NUM: ConfigProperty[String] = ConfigProperty


please move this config under HIVE_SYNC_AS_DATA_SOURCE_TABLE to make hive_sync config continuous.

leesf · 2021-07-02T15:19:18Z

hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java

@@ -104,6 +104,9 @@
  @Parameter(names = {"--decode-partition"}, description = "Decode the partition value if the partition has encoded during writing")
  public Boolean decodePartition = false;

+  @Parameter(names = {"--batch-sync-num"}, description = "how many partitions one batch when synchronous partitions to hive")
+  public String batchSyncNum = "1000";


Use Integer?

leesf · 2021-07-02T15:26:04Z

hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java

-  private String constructAddPartitions(String tableName, List<String> partitions) {
+  private List<String> constructAddPartitions(String tableName, List<String> partitions) {
+    List<String> result = new ArrayList<>();
+    int batchSyncPartitionNum = Integer.parseInt(syncConfig.batchSyncNum);


check batchSyncNum > 0?

leesf · 2021-07-02T15:30:39Z

hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java

      alterSQL.append("  PARTITION (").append(partitionClause).append(") LOCATION '").append(fullPartitionPath)
          .append("' ");
+      if ((i + 1) % batchSyncPartitionNum == 0) {
+        result.add(alterSQL.toString());


see duplicate lines in line#186, would we encapsulate a method such as getAlterTablePrefix ?

leesf · 2021-07-04T03:25:56Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala

@@ -462,6 +462,11 @@ object DataSourceWriteOptions {
    .defaultValue(false)
    .withDocumentation("Whether to sync the table as managed table.")

+  val HIVE_BATCH_SYNC_PARTITION_NUM: ConfigProperty[String] = ConfigProperty


ConfigProperty[String] -> ConfigProperty[Integer]

oh, i see many boolean type config properties in DataSourceOptions.scala are set to be ConfigProperty[String], so i follow this style, make this config to String

in scala, it's better to set it Int instead of Integer. fix the comment.

leesf

@xiarixiaoyao Just left one more minor comment, then should be good to land.

…ive MetaStore

…metastore to avoid oom problem (apache#3209)

change the insret overwrte return type [HUDI-1860] Test wrapper for insert_overwrite and insert_overwrite_table [HUDI-2084] Resend the uncommitted write metadata when start up (apache#3168) Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com> [HUDI-2081] Move schema util tests out from TestHiveSyncTool (apache#3166) [HUDI-2094] Supports hive style partitioning for flink writer (apache#3178) [HUDI-2097] Fix Flink unable to read commit metadata error (apache#3180) [HUDI-2085] Support specify compaction paralleism and compaction target io for flink batch compaction (apache#3169) [HUDI-2092] Fix NPE caused by FlinkStreamerConfig#writePartitionUrlEncode null value (apache#3176) [HUDI-2006] Adding more yaml templates to test suite (apache#3073) [HUDI-2103] Add rebalance before index bootstrap (apache#3185) Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com> [HUDI-1944] Support Hudi to read from committed offset (apache#3175) * [HUDI-1944] Support Hudi to read from committed offset * [HUDI-1944] Adding group option to KafkaResetOffsetStrategies * [HUDI-1944] Update Exception msg [HUDI-2052] Support load logFile in BootstrapFunction (apache#3134) Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com> [HUDI-89] Add configOption & refactor all configs based on that (apache#2833) Co-authored-by: Wenning Ding <wenningd@amazon.com> [MINOR] Update .asf.yaml to codify notification settings, turn on jira comments, gh discussions (apache#3164) - Turn on comment for jira, so we can track PR activity better - Create a notification settings that match https://gitbox.apache.org/schemes.cgi?hudi - Try and turn on "discussions" on Github, to experiment [MINOR] Fix broken build due to FlinkOptions (apache#3198) [HUDI-2088] Missing Partition Fields And PreCombineField In Hoodie Properties For Table Written By Flink (apache#3171) [MINOR] Add Documentation to KEYGENERATOR_TYPE_PROP (apache#3196) [HUDI-2105] Compaction Failed For MergeInto MOR Table (apache#3190) [HUDI-2051] Enable Hive Sync When Spark Enable Hive Meta For Spark Sql (apache#3126) [HUDI-2112] Support reading pure logs file group for flink batch reader after compaction (apache#3202) [HUDI-2114] Spark Query MOR Table Written By Flink Return Incorrect Timestamp Value (apache#3208) [HUDI-2121] Add operator uid for flink stateful operators (apache#3212) [HUDI-2123] Exception When Merge With Null-Value Field (apache#3214) [HUDI-2124] A Grafana dashboard for HUDI. (apache#3216) [HUDI-2057] CTAS Generate An External Table When Create Managed Table (apache#3146) [HUDI-1930] Bootstrap support configure KeyGenerator by type (apache#3170) * [HUDI-1930] Bootstrap support configure KeyGenerator by type [HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem (apache#3209) [HUDI-2126] The coordinator send events to write function when there are no data for the checkpoint (apache#3219) [HUDI-2127] Initialize the maxMemorySizeInBytes in log scanner (apache#3220) [HUDI-2058]support incremental query for insert_overwrite_table/insert_overwrite operation on cow table (apache#3139) [HUDI-2129] StreamerUtil.medianInstantTime should return a valid date time string (apache#3221) [HUDI-2131] Exception Throw Out When MergeInto With Decimal Type Field (apache#3224) [HUDI-2122] Improvement in packaging insert into smallfiles (apache#3213) [HUDI-2132] Make coordinator events as POJO for efficient serialization (apache#3223) [HUDI-2106] Fix flink batch compaction bug while user don't set compaction tasks (apache#3192) [HUDI-2133] Support hive1 metadata sync for flink writer (apache#3225) [HUDI-2089]fix the bug that metatable cannot support non_partition table (apache#3182) [HUDI-2028] Implement RockDbBasedMap as an alternate to DiskBasedMap in ExternalSpillableMap (apache#3194) Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local> [HUDI-2135] Add compaction schedule option for flink (apache#3226) [HUDI-2055] Added deltastreamer metric for time of lastSync (apache#3129) [HUDI-2046] Loaded too many classes like sun/reflect/GeneratedSerializationConstructorAccessor in JVM metaspace (apache#3121) Loaded too many classes when use kryo of spark to hudi Co-authored-by: weiwei.duan <weiwei.duan@linkflowtech.com> [HUDI-1996] Adding functionality to allow the providing of basic auth creds for confluent cloud schema registry (apache#3097) * adding support for basic auth with confluent cloud schema registry [HUDI-2093] Fix empty avro schema path caused by duplicate parameters (apache#3177) * [HUDI-2093] Fix empty avro schema path caused by duplicate parameters * rename shcmea option key * fix doc * rename var name [HUDI-2113] Fix integration testing failure caused by sql results out of order (apache#3204) [HUDI-2016] Fixed bootstrap of Metadata Table when some actions are in progress. (apache#3083) Metadata Table cannot be bootstrapped when any action is in progress. This is detected by the presence of inflight or requested instants. The bootstrapping is initiated in preWrite and postWrite of each commit. So bootstrapping will be retried again until it succeeds. Also added metrics for when the bootstrapping fails or a table is re-bootstrapped. This will help detect tables which are not getting bootstrapped. [HUDI-2140] Fixed the unit test TestHoodieBackedMetadata.testOnlyValidPartitionsAdded. (apache#3234) [HUDI-2115] FileSlices in the filegroup is not descending by timestamp (apache#3206) [HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows (apache#3149) [HUDI-2069] Refactored String constants (apache#3172) [HUDI-1105] Adding dedup support for Bulk Insert w/ Rows (apache#2206) [HUDI-2134]Add generics to avoif forced conversion in BaseSparkCommitActionExecutor#partition (apache#3232) [HUDI-2009] Fixing extra commit metadata in row writer path (apache#3075) [HUDI-2099]hive lock which state is WATING should be released, otherwise this hive lock will be locked forever (apache#3186) [MINOR] Fix build broken from apache#3186 (apache#3245) [HUDI-2136] Fix conflict when flink-sql-connector-hive and hudi-flink-bundle are both in flink lib (apache#3227) [HUDI-2087] Support Append only in Flink stream (apache#3174) Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com> UnitTest for deltaSync Removing cosmetic changes and reuse function for insert_overwrite_table unit test intial unit test for the insert_overwrite and insert_over_write_table Adding failed test code for insert_overwrite Revert "[HUDI-2087] Support Append only in Flink stream (apache#3174)" (apache#3251) This reverts commit 3715267. [HUDI-2147] Remove unused class AvroConvertor in hudi-flink (apache#3243) [MINOR] Fix some wrong assert reasons (apache#3248) [HUDI-2087] Support Append only in Flink stream (apache#3252) Co-authored-by: 喻兆靖 <yuzhaojing@bilibili.com> [HUDI-2143] Tweak the default compaction target IO to 500GB when flink async compaction is off (apache#3238) [HUDI-2142] Support setting bucket assign parallelism for flink write task (apache#3239) [HUDI-1483] Support async clustering for deltastreamer and Spark streaming (apache#3142) - Integrate async clustering service with HoodieDeltaStreamer and HoodieStreamingSink - Added methods in HoodieAsyncService to reuse code [HUDI-2107] Support Read Log Only MOR Table For Spark (apache#3193) [HUDI-2144]Bug-Fix:Offline clustering(HoodieClusteringJob) will cause insert action losing data (apache#3240) * fixed * add testUpsertPartitionerWithSmallFileHandlingAndClusteringPlan ut * fix CheckStyle Co-authored-by: yuezhang <yuezhang@freewheel.tv> [MINOR] Fix EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION config (apache#3250) [HUDI-2168] Fix for AccessControlException for anonymous user (apache#3264) [HUDI-2045] Support Read Hoodie As DataSource Table For Flink And DeltaStreamer test with insert-overwrite and insert-overwrite-table removing hardcoded action to pass the unit test [HUDI-1969] Support reading logs for MOR Hive rt table (apache#3033) [HUDI-2171] Add parallelism conf for bootstrap operator using delta-commit for insert_overwrite

…metastore to avoid oom problem (apache#3209)

wangxianghu changed the title ~~[HUDI-2116]sync 10w partitions to hive by using HiveSyncTool lead to …~~ [HUDI-2116] Sync 10w partitions to hive by using HiveSyncTool lead to … Jul 1, 2021

xiarixiaoyao force-pushed the batchSync branch from f4c7b37 to fbcd406 Compare July 2, 2021 03:58

xiarixiaoyao changed the title ~~[HUDI-2116] Sync 10w partitions to hive by using HiveSyncTool lead to …~~ [HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem Jul 2, 2021

leesf reviewed Jul 2, 2021

View reviewed changes

leesf self-assigned this Jul 2, 2021

xiarixiaoyao force-pushed the batchSync branch 3 times, most recently from e751390 to 27bb61a Compare July 3, 2021 15:53

leesf reviewed Jul 4, 2021

View reviewed changes

[HUDI-2116] support batch sync for HiveSyncTool to avoid the oom of h…

c1050df

…ive MetaStore

xiarixiaoyao force-pushed the batchSync branch from 27bb61a to c1050df Compare July 4, 2021 04:41

fix description

56f820d

leesf approved these changes Jul 4, 2021

View reviewed changes

leesf merged commit 6a71412 into apache:master Jul 4, 2021

ghost pushed a commit to shivagowda/hudi that referenced this pull request Jul 14, 2021

[HUDI-2116] Support batch synchronization of partition datas to hive …

fa8e950

…metastore to avoid oom problem (apache#3209)

ghost pushed a commit to shivagowda/hudi that referenced this pull request Jul 31, 2021

[HUDI-2116] Support batch synchronization of partition datas to hive …

8128bf9

…metastore to avoid oom problem (apache#3209)

xiarixiaoyao deleted the batchSync branch December 3, 2021 02:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem #3209

[HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem #3209

xiarixiaoyao commented Jul 1, 2021

hudi-bot commented Jul 1, 2021 •

edited

Loading

codecov-commenter commented Jul 1, 2021 •

edited

Loading

yanghua commented Jul 2, 2021

xiarixiaoyao commented Jul 2, 2021

leesf Jul 2, 2021 •

edited

Loading

leesf Jul 2, 2021

leesf Jul 2, 2021

leesf Jul 2, 2021

leesf Jul 4, 2021

xiarixiaoyao Jul 4, 2021 •

edited

Loading

leesf left a comment •

edited

Loading

[HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem #3209

[HUDI-2116] Support batch synchronization of partition datas to hive metastore to avoid oom problem #3209

Conversation

xiarixiaoyao commented Jul 1, 2021

Tips

What is the purpose of the pull request

Verify this pull request

Committer checklist

hudi-bot commented Jul 1, 2021 • edited Loading

CI report:

codecov-commenter commented Jul 1, 2021 • edited Loading

Codecov Report

yanghua commented Jul 2, 2021

xiarixiaoyao commented Jul 2, 2021

leesf Jul 2, 2021 • edited Loading

Choose a reason for hiding this comment

leesf Jul 2, 2021

Choose a reason for hiding this comment

leesf Jul 2, 2021

Choose a reason for hiding this comment

leesf Jul 2, 2021

Choose a reason for hiding this comment

leesf Jul 4, 2021

Choose a reason for hiding this comment

xiarixiaoyao Jul 4, 2021 • edited Loading

Choose a reason for hiding this comment

leesf left a comment • edited Loading

Choose a reason for hiding this comment

hudi-bot commented Jul 1, 2021 •

edited

Loading

codecov-commenter commented Jul 1, 2021 •

edited

Loading

leesf Jul 2, 2021 •

edited

Loading

xiarixiaoyao Jul 4, 2021 •

edited

Loading

leesf left a comment •

edited

Loading