Skip to content

A pgcolumnar.analyze() could read one column instead of all, and take null_frac and bounds exactly from zone maps #414

Description

@jdatcmd

The problem

ANALYZE on a wide columnar table decodes every column, because the table-AM analyze
callbacks are never told which columns were asked for. Measured on 300,000 rows over 20
columns (one int, nineteen text of 80 bytes), analysing one column:

columnar heap
ANALYZE w (k) 622 ms 92 ms

6.8x slower on the shape columnar storage should win.

The cheap fix does not work, and this is worth recording

The obvious repair is to make the analyze slot lazy, reusing the deferred-slot
machinery already used for index fetch, and let ANALYZE's own per-column access drive
the projection. It buys nothing. acquire_sample_rows does:

rows[numrows++] = ExecCopySlotHeapTuple(slot);      /* analyze.c:1273, and again at 1294 */

It copies the whole tuple into a heap tuple and computes per-column statistics
afterwards, so every column is materialised at sample time regardless. Knowing the
column list would not help either: even ANALYZE w (k) samples full tuples. That is
a core design, not an oversight, and it is why an AM-side projection cannot fix this.

What a function could do that core cannot

Two things, and the second is the more interesting one.

1. Read one column at a time. Statistics are per attribute. A function can scan
each analysed column independently, which is exactly the access pattern the storage is
built for.

2. Take some statistics exactly, from metadata, without reading data at all. Zone
maps already hold per chunk null_count, value_count, and min/max. So:

  • null_frac is exact, not sampled;
  • the low and high bounds a range estimate depends on are exact;
  • n_distinct and most_common_vals still need sampling, so this is a hybrid rather
    than a replacement.

That is better statistics, not merely cheaper ones, and it is a genuine advantage of
columnar storage rather than a workaround.

Writing the results is supported on 18+, and not before

Checked on all three majors rather than assumed:

function PG17 PG18 PG19
pg_restore_attribute_stats absent present present
pg_restore_relation_stats absent present present
pg_clear_attribute_stats absent present present

So on 18 and later this writes statistics through a documented API. On 15 to 17 it
would mean writing pg_statistic directly, which is a materially different risk
(stakind slot layout, stavalues typing, staop/stacoll correctness) and a real
version-support decision rather than a detail. My inclination is 18+ only, and to
say so plainly rather than ship a fragile catalog writer for older majors.

The catch that should decide this

Autoanalyze will not call it. Same shape as pgcolumnar.vacuum(), but the
consequence differs in kind: a stale vacuum wastes space, while stale statistics
silently produce bad plans. We would be handing users a fast path that quietly rots,
and the failure mode is a query that got slower for no visible reason.

See #415, which proposes a background worker to schedule exactly this class of work.
If that lands, this becomes considerably more attractive, because the scheduling
gap is the main objection.

What I would do first, before building anything

Measure whether scan_analyze_next_tuple itself can be made competitive, because
that is the path autoanalyze actually takes. The slice-based group sampling is already
there; the open question is whether 622 ms against heap's 92 ms is inherent to decoding
whole groups for scattered sampled rows, or whether there is slack in it.

That measurement decides what this issue is. If the AM path can be brought close to
heap, a separate function is a maintenance burden with a stale-statistics footgun
attached. If it cannot, the function is the only route to statistics that match what
the storage can cheaply provide.

Proposed shape, if it goes ahead

pgcolumnar.analyze(rel regclass, columns text[] DEFAULT NULL)

Core ANALYZE stays the correctness path and keeps working with autoanalyze. This is
an opt-in accelerator for wide tables, documented as needing scheduling, exactly as
pgcolumnar.vacuum() is.

Found while answering how the table-AM interface's lack of projection support is
solved. See also #413, which is the same root cause on the index-build path and is
fixable, because that callback is told which columns it needs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions