From ebdd5486e309e606efa1ddeb7d2b6d62f2315541 Mon Sep 17 00:00:00 2001 From: shardul-cr7 Date: Thu, 13 Dec 2018 14:12:18 +0530 Subject: [PATCH] [CARBONDATA-3166]Updated Document and added Column Compressor used in Describe Formatted Updated Document and added column compressor used in Describe Formatted Command This closes #2986 --- docs/configuration-parameters.md | 2 +- .../command/table/CarbonDescribeFormattedCommand.scala | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md index 4aa29295aa6..db21c6ae062 100644 --- a/docs/configuration-parameters.md +++ b/docs/configuration-parameters.md @@ -91,7 +91,7 @@ This section provides the details of all the configurations required for the Car | carbon.dictionary.server.port | 2030 | Single Pass Loading enables single job to finish data loading with dictionary generation on the fly. It enhances performance in the scenarios where the subsequent data loading after initial load involves fewer incremental updates on the dictionary. Single pass loading can be enabled using the option ***carbon.options.single.pass***. When this option is specified, a dictionary server will be internally started to handle the dictionary generation and query requests. This configuration specifies the port on which the server need to listen for incoming requests. Port value ranges between 0-65535 | | carbon.load.directWriteToStorePath.enabled | false | During data load, all the carbondata files are written to local disk and finally copied to the target store location in HDFS/S3. Enabling this parameter will make carbondata files to be written directly onto target HDFS/S3 location bypassing the local disk.**NOTE:** Writing directly to HDFS/S3 saves local disk IO(once for writing the files and again for copying to HDFS/S3) there by improving the performance. But the drawback is when data loading fails or the application crashes, unwanted carbondata files will remain in the target HDFS/S3 location until it is cleared during next data load or by running *CLEAN FILES* DDL command | | carbon.options.serialization.null.format | \N | Based on the business scenarios, some columns might need to be loaded with null values. As null value cannot be written in csv files, some special characters might be adopted to specify null values. This configuration can be used to specify the null values format in the data being loaded. | -| carbon.column.compressor | snappy | CarbonData will compress the column values using the compressor specified by this configuration. Currently CarbonData supports 'snappy' and 'zstd' compressors. | +| carbon.column.compressor | snappy | CarbonData will compress the column values using the compressor specified by this configuration. Currently CarbonData supports 'snappy', 'zstd' and 'gzip' compressors. | | carbon.minmax.allowed.byte.count | 200 | CarbonData will write the min max values for string/varchar types column using the byte count specified by this configuration. Max value is 1000 bytes(500 characters) and Min value is 10 bytes(5 characters). **NOTE:** This property is useful for reducing the store size thereby improving the query performance but can lead to query degradation if value is not configured properly. | | ## Compaction Configuration diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala index 151359e9077..2d560dfa3d1 100644 --- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala +++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala @@ -92,7 +92,9 @@ private[sql] case class CarbonDescribeFormattedCommand( Strings.formatSize( tblProps.getOrElse(CarbonCommonConstants.CARBON_LOAD_MIN_SIZE_INMB, CarbonCommonConstants.CARBON_LOAD_MIN_SIZE_INMB_DEFAULT).toFloat), ""), - + ("Data File Compressor ", tblProps + .getOrElse(CarbonCommonConstants.COMPRESSOR, + CarbonCommonConstants.DEFAULT_COMPRESSOR), ""), ////////////////////////////////////////////////////////////////////////////// // Index Information //////////////////////////////////////////////////////////////////////////////