[SPARK-20187][SQL] Replace loadTable with moveFile to speed up load table for many output files #17505
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #75445 has finished for PR 17505 at commit
|
rxin
reviewed
Apr 4, 2017
| private lazy val moveFileMethod = | ||
| findMethod( | ||
| classOf[Hive], | ||
| "moveFile", |
There was a problem hiding this comment.
does this exist in all the versions Spark supports?
There was a problem hiding this comment.
moveFile is supported from Hive-1.1.0.
gatorsmile
reviewed
Apr 4, 2017
| isSrcLocal) | ||
| val tbl = client.getTable(tableName) | ||
| val fs = tbl.getDataLocation.getFileSystem(conf) | ||
| if (replace) { |
There was a problem hiding this comment.
loadTable is calling replaceFiles when replace is true. , instead of calling Hive.copyFiles.
replaceFiles is based on the calls of moveFile.
Feel free to let me know if anything is missing.
There was a problem hiding this comment.
It sounds like your PR is similar to what Hive did in HIVE-12908 . Right?
There was a problem hiding this comment.
replaceFiles will move files one by one, so it will take a long time if load many output files, as follows:
17/04/01 10:50:04 INFO TaskSetManager: Finished task 216432.0 in stage 0.0 (TID 216866) in 3022 ms on jqhadoop-test28-52.int.yihaodian.com (executor 87) (216868/216869)
17/04/01 10:50:04 INFO TaskSetManager: Finished task 207165.0 in stage 0.0 (TID 216796) in 5952 ms on jqhadoop-test28-8.int.yihaodian.com (executor 54) (216869/216869)
17/04/01 10:50:04 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/04/01 10:50:04 INFO DAGScheduler: ResultStage 0 (processCmd at CliDriver.java:376) finished in 541.797 s
17/04/01 10:50:04 INFO DAGScheduler: Job 0 finished: processCmd at CliDriver.java:376, took 551.208919 s
17/04/01 10:50:04 INFO FileFormatWriter: Job null committed.
17/04/01 10:50:14 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-00000-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-00000-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 10:50:14 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-00001-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-00001-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 10:50:14 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-00002-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-00002-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 10:50:14 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-00003-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-00003-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 10:50:14 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-00004-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-00004-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 10:50:14 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-00005-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-00005-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 10:50:14 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-00006-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-00006-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 10:50:14 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-00007-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-00007-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 10:50:14 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-00008-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-00008-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
...
17/04/01 11:16:11 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-99995-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-99995-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 11:16:11 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-99996-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-99996-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 11:16:11 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-99997-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-99997-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 11:16:11 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-99998-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-99998-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 11:16:11 INFO Hive: Replacing src:viewfs://cluster4/user/hive/warehouse/staging/.hive-staging_hive_2017-04-01_10-40-02_349_8047899863313770218-1/-ext-10000/part-99999-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, dest: viewfs://cluster4/user/hive/warehouse/tmp.db/spark_load_slow/part-99999-9335c5f3-60fa-418b-a466-2d76a5e84537-c000, Status:true
17/04/01 11:16:18 INFO SparkSqlParser: Parsing command: `tmp`.`spark_load_slow`
17/04/01 11:16:18 INFO CatalystSqlParser: Parsing command: string
17/04/01 11:16:18 INFO CatalystSqlParser: Parsing command: string
17/04/01 11:16:18 INFO CatalystSqlParser: Parsing command: string
17/04/01 11:16:18 INFO CatalystSqlParser: Parsing command: string
17/04/01 11:16:18 INFO CatalystSqlParser: Parsing command: string
Time taken: 2178.736 seconds
17/04/01 11:16:18 INFO CliDriver: Time taken: 2178.736 seconds
moveFile only move once.
Full log can be find spark.loadTable.log.tar.gz, SPARK-20187.
There was a problem hiding this comment.
This PR has nothing to do with dynamic partition.
There was a problem hiding this comment.
If you read the changes made in that PR, it is doing the similar things.
… warehouse/dbName.db/.hive-staging
|
Test build #75535 has finished for PR 17505 at commit
|
|
@gatorsmile You are right. It is similar to HIVE-12908. There are 3 bottlenecks for many output files:
|
|
@wangyum @rxin @gatorsmile Is it possible to upgrade the built-in hive-exec to resolve this problem? I believe the built-in hive-exec is this https://github.com/JoshRosen/hive/tree/release-1.2.1-spark2 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
HiveClientImpl.loadTable load files one by one, so this step will take a long time if a job generates many files. This PR replace
Hive.loadTablewithHive.moveFileto speed up this step forcreate table tableName as select ...andinsert overwrite table tableName select ....How was this patch tested?
manual tests