[SPARK-39543] The option of DataFrameWriterV2 should be passed to storage properties if fallback to v1 #36941
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Could you please take a look when you have a time, thanks @cloud-fan |
cloud-fan
reviewed
Jun 21, 2022
| @@ -531,6 +532,15 @@ class DataFrameWriterV2Suite extends QueryTest with SharedSparkSession with Befo | |||
| assert(table.properties === (Map("provider" -> "foo") ++ defaultOwnership).asJava) | |||
| } | |||
|
|
|||
| test("SPARK-39543 writeOption should be passed to storage properties when fallback to v1") { | |||
| spark.range(10).writeTo("table_name").option("compression", "zstd").using("parquet").create() | |||
There was a problem hiding this comment.
can we test with InMemoryV1Provider? We may migrate the file source to v2 completely in the future and then this test won't test the v1 fallback anymore.
cloud-fan
reviewed
Jun 22, 2022
| test("SPARK-39543 writeOption should be passed to storage properties when fallback to v1") { | ||
| val provider = classOf[InMemoryV1Provider].getName | ||
| val oldConf = spark.sessionState.conf.getConf(SQLConf.USE_V1_SOURCE_LIST) | ||
| try { |
There was a problem hiding this comment.
nit: we can use withSQLConf
cloud-fan
approved these changes
Jun 22, 2022
|
Can one of the admins verify this patch? |
cloud-fan
reviewed
Jun 22, 2022
| test("SPARK-39543 writeOption should be passed to storage properties when fallback to v1") { | ||
| val provider = classOf[InMemoryV1Provider].getName | ||
|
|
||
| withSQLConf((SQLConf.USE_V1_SOURCE_LIST.key, provider)) { |
There was a problem hiding this comment.
looking at other tests that use InMemoryV1Provider, it seems we don't need to set USE_V1_SOURCE_LIST to trigger v1 fallback?
There was a problem hiding this comment.
Yea, Other tests trigger v1 fallback without set USE_V1_SOURCE_LIST , AFAIK,
- Other tests aim to test the read/write process, and the
InMemoryV1Provideris actually a v2 format, and we trigger v1 fallback at thenewScanBuilder&newWriteBuilderlayer. - This test in PR needs to be fallback to V1 when the table is created, so we need to set
USE_V1_SOURCE_LIST(see: isV2Provider)
|
thanks, merging to master/3.3/3.2! |
cloud-fan
pushed a commit
that referenced
this pull request
Jun 23, 2022
…rage properties if fallback to v1
### What changes were proposed in this pull request?
The option of DataFrameWriterV2 should be passed to storage properties if fallback to v1, to support something such as compressed formats
### Why are the changes needed?
example:
`spark.range(0, 100).writeTo("t1").option("compression", "zstd").using("parquet").create`
**before**
gen: part-00000-644a65ed-0e7a-43d5-8d30-b610a0fb19dc-c000.**snappy**.parquet ...
**after**
gen: part-00000-6eb9d1ae-8fdb-4428-aea3-bd6553954cdd-c000.**zstd**.parquet ...
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
new test
Closes #36941 from Yikf/writeV2option.
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit e5b7fb8)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
cloud-fan
pushed a commit
that referenced
this pull request
Jun 23, 2022
…rage properties if fallback to v1
The option of DataFrameWriterV2 should be passed to storage properties if fallback to v1, to support something such as compressed formats
example:
`spark.range(0, 100).writeTo("t1").option("compression", "zstd").using("parquet").create`
**before**
gen: part-00000-644a65ed-0e7a-43d5-8d30-b610a0fb19dc-c000.**snappy**.parquet ...
**after**
gen: part-00000-6eb9d1ae-8fdb-4428-aea3-bd6553954cdd-c000.**zstd**.parquet ...
No
new test
Closes #36941 from Yikf/writeV2option.
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit e5b7fb8)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
sunchao
pushed a commit
to sunchao/spark
that referenced
this pull request
Jun 2, 2023
…rage properties if fallback to v1
The option of DataFrameWriterV2 should be passed to storage properties if fallback to v1, to support something such as compressed formats
example:
`spark.range(0, 100).writeTo("t1").option("compression", "zstd").using("parquet").create`
**before**
gen: part-00000-644a65ed-0e7a-43d5-8d30-b610a0fb19dc-c000.**snappy**.parquet ...
**after**
gen: part-00000-6eb9d1ae-8fdb-4428-aea3-bd6553954cdd-c000.**zstd**.parquet ...
No
new test
Closes apache#36941 from Yikf/writeV2option.
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit e5b7fb8)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The option of DataFrameWriterV2 should be passed to storage properties if fallback to v1, to support something such as compressed formats
Why are the changes needed?
example:
spark.range(0, 100).writeTo("t1").option("compression", "zstd").using("parquet").createbefore
gen: part-00000-644a65ed-0e7a-43d5-8d30-b610a0fb19dc-c000.snappy.parquet ...
after
gen: part-00000-6eb9d1ae-8fdb-4428-aea3-bd6553954cdd-c000.zstd.parquet ...
Does this PR introduce any user-facing change?
No
How was this patch tested?
new test