Skip to content

Late materialization details

Jump to bottom
Alexey Ozeritskiy edited this page Oct 14, 2026 · 1 revision

Late materialization settings and lookup

Implementation reference checked against f5e786f. Fundamental explanations remain in docs/arch.

Row Locator

The row locator is one u64 value. The high 32 bits store the Parquet row group id. The low 32 bits store the row offset in that group.

The name __row_id__ is reserved. A Parquet file cannot have a column with this name. A SQL query cannot use it as an output name.

Planner Rule

The rule is on by default. It changes a plan only when all these checks pass:

  • The plan has one Parquet source.
  • Between the source and the limit, filters keep the same row identity.
  • The final output has direct source columns, with no computed expressions.
  • The plan has a known positive limit. The default maximum is 100 rows.
  • The eager scan estimate is at least two times the late estimate.

The cost model adds the narrow scan bytes and the lookup bytes. It gets compressed and uncompressed sizes from Parquet metadata. It uses 1 MiB as an estimated page size. It also computes an upper bound from full row-group chunks. The lower value becomes the lookup estimate. This estimate is not an exact count of bytes from disk.

Parquet Lookup

The source resolves column names and types once for each lookup task. Each task gets a read-only physical row reader. The readers share the FileReader that the source has already opened. QDB does not open the file again.

For supported flat primitive and string columns, the reader first tries the page path. This path reads only pages that contain requested rows. Some column types and page layouts cannot use this path. In that case, the reader loads the affected row groups and then selects the rows. Both paths keep column order and duplicate column names.

If the page path starts, only Invalid and NotImplemented can cause a move to the row-group path. An I/O error or a memory error stops the query.

The shared FileReader depends on three current rules:

  • The input is an Arrow ReadableFile.
  • Arrow pre-buffer is off.
  • The lookup code does not use page-index or bloom-filter readers.

Code that changes one of these rules must also check shared access again.

Logical Plan

The logical operator stores its input, locator name, and column map. It does not store a live source pointer. After an S-expression parse, SourceFactory provides the source. The plan pipeline checks that the operator has one lookup source and then turns row locators on.

CLI and Browser

These CLI options control the rule:

--late-materialization=auto|off
--late-materialization-max-rows <n>
--late-materialization-min-savings <factor>

auto is the default. The default maximum is 100 rows. The default minimum cost factor is 2.0. With --verbose, QDB prints the planner decision and its byte estimates.

The browser runtime has no source lookup task. Plan export turns late materialization off for browser plans.

Clone this wiki locally