Skip to content

Commit

Permalink
[SPARK-34863][SQL][FOLLOWUP] Disable `spark.sql.parquet.enableNestedC…
Browse files Browse the repository at this point in the history
…olumnVectorizedReader` by default

### What changes were proposed in this pull request?

This PR disables `spark.sql.parquet.enableNestedColumnVectorizedReader` by default.

### Why are the changes needed?

In apache#34659 the config was turned mainly for testing reason. As the feature is new, we should turn it off by default.

### Does this PR introduce _any_ user-facing change?

The config `spark.sql.parquet.enableNestedColumnVectorizedReader` is turned off by default now.

### How was this patch tested?

Existing tests.

Closes apache#36055 from sunchao/disable.

Authored-by: Chao Sun <sunchao@apple.com>
Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
  • Loading branch information
sunchao authored and viirya committed Apr 4, 2022
1 parent fbcab01 commit 1b08673
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 5 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -1015,7 +1015,7 @@ object SQLConf {
s"Requires ${PARQUET_VECTORIZED_READER_ENABLED.key} to be enabled.")
.version("3.3.0")
.booleanConf
.createWithDefault(true)
.createWithDefault(false)

val PARQUET_RECORD_FILTER_ENABLED = buildConf("spark.sql.parquet.recordLevelFilter.enabled")
.doc("If true, enables Parquet's native record-level filtering using the pushed down " +
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1125,8 +1125,7 @@ struct<plan:string>
-- !query output
== Physical Plan ==
*Filter v#x IN ([a],null)
+- *ColumnarToRow
+- FileScan parquet default.t[v#x] Batched: true, DataFilters: [v#x IN ([a],null)], Format: Parquet, Location [not included in comparison]/{warehouse_dir}/t], PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema: struct<v:array<string>>
+- FileScan parquet default.t[v#x] Batched: false, DataFilters: [v#x IN ([a],null)], Format: Parquet, Location [not included in comparison]/{warehouse_dir}/t], PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema: struct<v:array<string>>


-- !query
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1067,8 +1067,7 @@ struct<plan:string>
-- !query output
== Physical Plan ==
*Filter v#x IN ([a],null)
+- *ColumnarToRow
+- FileScan parquet default.t[v#x] Batched: true, DataFilters: [v#x IN ([a],null)], Format: Parquet, Location [not included in comparison]/{warehouse_dir}/t], PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema: struct<v:array<string>>
+- FileScan parquet default.t[v#x] Batched: false, DataFilters: [v#x IN ([a],null)], Format: Parquet, Location [not included in comparison]/{warehouse_dir}/t], PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema: struct<v:array<string>>


-- !query
Expand Down

0 comments on commit 1b08673

Please sign in to comment.