[SPARK-25356][SQL]Add Parquet block size option to SparkSQL configuration #22350
Conversation
| @@ -123,6 +123,9 @@ class ParquetFileFormat | |||
| // Sets compression scheme | |||
| conf.set(ParquetOutputFormat.COMPRESSION, parquetOptions.compressionCodecClassName) | |||
|
|
|||
| // Sets Parquet block size | |||
| conf.setInt(ParquetOutputFormat.BLOCK_SIZE, sparkSession.sessionState.conf.parquetBlockSize) | |||
There was a problem hiding this comment.
For clarification, we are already able to set this via parquet.block.size but this PR proposes an alias for it, right?
There was a problem hiding this comment.
Yes, we are already able to set this via parquet.block.size,
I think we should add this parameter into "sql-programming-guide.md"
There was a problem hiding this comment.
I doubt if it is common enough to have an alias and document this in sql-programming-guide.md. Other configurations like parquet.page.size, parquet.enable.dictionary or parquet.writer.version are also rather similarly used as much as that configuration in my experience.
I wouldn't add this for now.
There was a problem hiding this comment.
Sounds reasonable. I close it now， thanks
|
Test build #95752 has finished for PR 22350 at commit
|
|
Sorry its not clear what is the setting - how do we adjust this setting, I wish to adjust parquet block size. |
|
@AnirudhVyas, set |
What changes were proposed in this pull request?
I think we should configure the Parquet buffer size when using Parquet format.
Because for HDFS,
dfs.block.sizeis configurable, sometimes we hope the block size of parquet to be consistent with it.And whether this parameter
spark.sql.files.maxPartitionBytesis best consistent with the Parquet block size when using Parquet format?Also we may want to shrink Parquet block size in some tests.
How was this patch tested?
N/A