Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-42595][SQL] Support query inserted partitions after insert data into table when hive.exec.dynamic.partition=true #40298

Closed
wants to merge 4 commits into from

Conversation

haoyanzhang
Copy link

What changes were proposed in this pull request?

Introduce spark.hive.exec.dynamic.partition.savePartitions=true (default false) spark.hive.exec.dynamic.partition.savePartitions.tableNamePrefix=hive_dynamic_inserted_partitions
when spark.hive.exec.dynamic.partition.savePartitions=true we save the partitions to the
temporary view $spark.hive.exec.dynamic.partition.savePartitions.tableNamePrefix_$dbName_$tableName

Why are the changes needed?

When hive.exec.dynamic.partition=true and hive.exec.dynamic.partition.mode=nonstrict, we can insert table by sql like 'insert overwrite table aaa partition(dt) select xxxx', of course we can know the partitions inserted into the table by the sql itself, but if we want do something for common use, we need some common way to get the inserted partitions, for example:

spark.sql("insert overwrite table aaa partition(dt) select xxxx")  //insert table
val partitions = getInsertedPartitions()   //need some way to get inserted partitions
monitorInsertedPartitions(partitions)    //do something for common use

This pull request will allow user to get inserted partitions in a common way

Does this PR introduce any user-facing change?

no

How was this patch tested?

new unit test

@github-actions github-actions bot added the SQL label Mar 6, 2023
@github-actions
Copy link

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

@github-actions github-actions bot added the Stale label Jun 15, 2023
@github-actions github-actions bot closed this Jun 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant