Skip to content

Commit

Permalink
match the doc in SQLConf with programming guide
Browse files Browse the repository at this point in the history
  • Loading branch information
seancxmao committed Sep 25, 2018
1 parent e6f67c1 commit 6e2680f
Showing 1 changed file with 10 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -474,8 +474,16 @@ object SQLConf {
.createWithDefault(10)

val PARQUET_WRITE_LEGACY_FORMAT = buildConf("spark.sql.parquet.writeLegacyFormat")
.doc("Whether to be compatible with the legacy Parquet format adopted by Spark 1.4 and prior " +
"versions, when converting Parquet schema to Spark SQL schema and vice versa.")
.doc("This configuration indicates whether we should use legacy Parquet format adopted by " +
"Spark 1.4 and prior versions or the standard format defined in parquet-format " +
"specification to write Parquet files. This is not only related to compatibility with old " +
"Spark ones, but also other systems like Hive, Impala, Presto, etc. This is especially " +
"important for decimals. If this configuration is not enabled, decimals will be written in " +
"int-based format in Spark 1.5 and above, other systems that only support legacy decimal " +
"format (fixed length byte array) will not be able to read what Spark has written. Note " +
"other systems may have added support for the standard format in more recent versions, " +
"which will make this configuration unnecessary. Please consult documentation of related " +
"systems for details.")
.booleanConf
.createWithDefault(false)

Expand Down

0 comments on commit 6e2680f

Please sign in to comment.