compression

https://www.sparkcodehub.com/spark-compression-techniques

Snappy

• parquet 格式的默认压缩算法
• 速度最快

Gzip

• 最高压缩率
• used for compressing text files and log files.

LZ4

• used for large data sets.

LZO

• 表现有时比snappy好
• 由于license的原因，没有预安装于Databricks
• for compressing Hadoop and Spark data.

Bzip2: Bzip2 is a compression codec that provides a high compression ratio but is relatively slow compared to other codecs. It is commonly used for compressing large text files and is a popular choice for archiving data.

Zlib: Zlib is a compression codec that provides a good balance between compression ratio and speed. It is commonly used for compressing text files and log files.

Deflate: Deflate is a compression codec that is widely used for compressing data in zip files. It provides a good compression ratio and is relatively fast compared to other codecs.