Add LZO compression support to gpfdist

[ZTE-EBASE]

Description

This PR adds LZO compressed file format read/write support to the gpfdist utility in Apache Cloudberry.
Key Insight: LZO is specifically designed for speed over compression ratio. It's ideal for scenarios where decompression throughput matters more than storage savings - such as large-scale data loading into MPP databases like Cloudberry.

Core Features

Read Support (.lzo → Cloudberry):

• ✅ Block-level decompression: Processes LZO data block by block, extracting uncompressed length and compressed length from each block header
• ✅ Checksum verification: Validates data integrity using Adler32 and CRC32 checksums embedded in LZO blocks
• ✅ Enhanced error reporting: Provides detailed, actionable error messages for various failure scenarios (truncated files, corrupt data, invalid format, checksum mismatches)

Write Support (Cloudberry → .lzo):

• ✅ Table data unloading: Exports table data to LZO-compressed external files
• ✅ Standard LZOP format output: Generates files compatible with standard lzop tool and Hadoop LZO readers
• ✅ Configurable compression level: Supports LZO compression levels (default: level 1, fastest; configurable up to level 9)

Future Potential - Hadoop Ecosystem Integration:

• 🔮 Direct query of Hive tables stored in LZO-compressed format on HDFS
• 🔮 Seamless integration with Spark LZO Parquet output
• 🔮 Accelerated data loading from Sqoop LZO imports

Use case/motivation

Scenario 1: High-Volume Data Loading
LZO is a lightweight compression algorithm optimized for decompression speed (~500MB/s), making it ideal for scenarios requiring rapid data ingestion. Compared to BZip2 (~10MB/s) and GZIP (~100MB/s), LZO provides the fastest decompression while maintaining a reasonable compression ratio.

Scenario 2: Hadoop Ecosystem Integration
Many Hadoop ecosystem tools natively support LZO compression:

1. Hive: Supports LZO-compressed text and sequence files
2. Spark: Can write LZO-compressed output (via hadoop-lzo library)
3. Sqoop: Commonly uses LZO for bulk imports from RDBMS to HDFS

References

Technical Documentation

• LZO Official Website
• LZOP File Format Specification
• Hadoop LZO Compression Guide

Cloudberry Documentation

• GPFDIST Official Docs
• External Tables Guide

Related Projects

• Apache Cloudberry GitHub
• Greenplum Database (upstream)

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[jianlirong]

Thanks for raising this question.

We'd like to better understand your use case.

Today, most modern data platforms have largely standardized on compression algorithms such as LZ4 and Zstandard (Zstd), which generally provide a better balance between compression ratio, decompression speed, and ecosystem support. As a result, they have become the default choice for many new deployments.

LZO certainly played an important role historically, especially in some Hadoop-based ecosystems, but we're curious about the motivation for introducing LZO support in Apache Cloudberry today.

Is your primary requirement:

• compatibility with an existing data lake or Hadoop environment?

• the ability to read legacy LZO-compressed data?

• or is there another workload where LZO provides a specific advantage?

Understanding the underlying use case would help us evaluate whether native LZO support is the right solution, or whether there might be a better alternative.

We're always happy to discuss new features, but we'd first like to understand the problem we're trying to solve rather than focusing on a particular implementation.

Lirong

[ZTE-EBASE]

Driven by requirements: In some production environments, ETL processes place LZO-compressed data lake files onto file servers, which are then imported into databases. I am fully aware of the compression characteristics of LZ4 and Zstd, but they cannot address the business requirements. Given this context, I have implemented support for LZO-compressed files in gpfdist, and I would like to contribute this feature to the Apache Cloudberry open-source community in the future.

ZTE-EBASE

[jianlirong]

Thanks for the clarification. That makes sense.

We understand now that the primary motivation is not to choose LZO over LZ4 or Zstd for new deployments, but to support existing production environments where LZO-compressed data is already part of the ETL and data lake workflow.

In that context, supporting LZO in gpfdist can be a useful compatibility feature for Cloudberry users who need to integrate with existing data pipelines and file-based data sources.

We also appreciate that you have already implemented LZO support in gpfdist and are willing to contribute it to the Apache Cloudberry community.

Please feel free to open a PR when the implementation is ready. We can then review the design, testing, documentation, and any compatibility considerations together.

Thanks again for taking the initiative to contribute to Cloudberry!

Lirong