Skip to content

Understanding Parquet Metadata

Jump to bottom
Miguel P Z edited this page Apr 4, 2026 · 1 revision

Understanding Parquet Metadata

Home

iparq is a tool for inspecting the metadata inside Apache Parquet files. This page explains the Parquet concepts that iparq exposes so you can better understand what the output means and why it matters.

What is Apache Parquet?

Apache Parquet is a columnar storage format designed for analytics and big data workloads. Instead of storing data row by row, it stores values for each column together, which makes it efficient for:

  • Reading only the columns a query needs
  • Compressing similar values very effectively
  • Supporting rich encodings and metadata for query optimization

Parquet is widely used across the modern data ecosystem, including tools such as Apache Spark, DuckDB, Pandas, Apache Arrow, Trino, and many cloud data platforms.

Parquet File Structure

At a high level, a Parquet file is organized like this:

  1. File metadata
  2. Row groups
  3. Column chunks
  4. Pages

File metadata

The file metadata lives in the Parquet footer and describes the file as a whole: schema, creator, row counts, row group counts, version information, and other details. This is the information represented by ParquetMetaModel in iparq.

Row groups

A row group is a horizontal partition of the dataset. Each row group contains a subset of rows for every column in the schema.

Row groups matter because they are often the unit of parallel reading and skipping. If a query engine can rule out a row group using statistics or bloom filters, it avoids reading that part of the file.

Column chunks

Within each row group, each column is stored in its own column chunk. So for every row group and every column, there is one corresponding column chunk.

This is where you see details such as:

  • compression codec
  • statistics
  • encryption metadata
  • sizes

Pages

Pages are the smallest unit of storage inside a column chunk. Data pages hold encoded values, and other page types can hold indexes or dictionaries.

Most end users do not inspect pages directly, but pages are where Parquet’s encoding and compression efficiency is actually realized.

What iparq Shows

File Metadata

iparq shows top-level file metadata such as:

  • created_by: Which engine or library created the file, for example parquet-cpp-arrow version 14.0.2, parquet-mr, or similar
  • num_columns: Number of columns in the schema
  • num_rows: Total row count in the file
  • num_row_groups: Number of row groups in the file; this can affect parallelism and skipping behavior
  • format_version: Parquet format version, such as 1.0, 2.4, or 2.6
  • serialized_size: Size of the metadata footer in bytes

These fields are often the fastest way to understand how a file was produced and how it is laid out.

Compression Codecs

Parquet stores data column by column, which makes compression especially effective. iparq shows the compression codec used for each column chunk.

Common codecs include:

  • SNAPPY: fast, widely supported, moderate compression ratio
  • ZSTD: excellent balance of compression ratio and speed
  • GZIP: high compression ratio, but slower
  • LZ4: very fast, often with lower compression than ZSTD or GZIP
  • BROTLI: strong compression, often used when size matters more than speed
  • UNCOMPRESSED: no compression at all

Different engines choose different defaults, and those choices can have a real impact on storage cost and query performance. In many ways, this question of “what compression is this Parquet file actually using?” is what motivated the creation of iparq.

Bloom Filters

A bloom filter is a probabilistic data structure used for fast membership testing.

In Parquet, bloom filters can help a query engine decide that a row group definitely does not contain a value, which allows it to skip reading that data entirely.

Important properties:

  • False positives are possible
  • False negatives are not

That means a bloom filter may sometimes say “maybe present” when the value is not there, but it should not say “definitely absent” when the value is actually present.

This can dramatically speed up selective lookups, especially for equality filters on large datasets.

Further reading: https://duckdb.org/2025/03/07/parquet-bloom-filters-in-duckdb.html

Encryption

Parquet supports column-level encryption. iparq detects encrypted columns by checking whether is_crypto_metadata_set() is true for a column chunk.

Encrypted columns are shown with a 🔒 indicator.

This is useful when auditing files, verifying security-sensitive pipelines, or understanding why certain metadata may be limited.

Min/Max Statistics

Parquet can store column statistics in metadata, including minimum and maximum values for a column chunk.

These statistics enable predicate pushdown. For example, if a row group’s max value for a date column is 2024-01-31, then a filter for dates in March can skip that row group without reading it.

iparq shows min/max values per column per row group so you can see what the query engine may use for skipping.

Statistics Exactness (PyArrow 22+)

Newer PyArrow versions expose:

  • is_min_value_exact
  • is_max_value_exact

These fields indicate whether the stored min/max values are exact or whether they may be truncated or approximate.

This matters because query optimizers need to know whether statistics are safe to rely on for pruning decisions. Exactness helps explain both correctness and optimization behavior.

Column Sizes and Compression Ratios

With the --sizes flag, iparq shows additional storage information such as:

  • num_values
  • total_compressed_size in human-readable form
  • compression ratio, computed as uncompressed size divided by compressed size

This helps you understand how efficiently each column is stored and which columns are driving space usage.

Why This Matters

These details are not just metadata trivia:

  • Different engines produce different Parquet files
  • Knowing compression helps optimize storage and performance
  • Bloom filters can dramatically speed up point lookups
  • Statistics enable predicate pushdown optimization
  • Understanding these details helps debug performance issues

If two tools produce very different file sizes or query speeds from the same data, Parquet metadata is often where the explanation starts.

Further Reading

Home

iparq Wiki

Links

Clone this wiki locally