The problem
ANALYZE on a wide columnar table decodes every column, because the table-AM analyze
callbacks are never told which columns were asked for. Measured on 300,000 rows over 20
columns (one int, nineteen text of 80 bytes), analysing one column:
|
columnar |
heap |
ANALYZE w (k) |
622 ms |
92 ms |
6.8x slower on the shape columnar storage should win.
The cheap fix does not work, and this is worth recording
The obvious repair is to make the analyze slot lazy, reusing the deferred-slot
machinery already used for index fetch, and let ANALYZE's own per-column access drive
the projection. It buys nothing. acquire_sample_rows does:
rows[numrows++] = ExecCopySlotHeapTuple(slot); /* analyze.c:1273, and again at 1294 */
It copies the whole tuple into a heap tuple and computes per-column statistics
afterwards, so every column is materialised at sample time regardless. Knowing the
column list would not help either: even ANALYZE w (k) samples full tuples. That is
a core design, not an oversight, and it is why an AM-side projection cannot fix this.
What a function could do that core cannot
Two things, and the second is the more interesting one.
1. Read one column at a time. Statistics are per attribute. A function can scan
each analysed column independently, which is exactly the access pattern the storage is
built for.
2. Take some statistics exactly, from metadata, without reading data at all. Zone
maps already hold per chunk null_count, value_count, and min/max. So:
null_frac is exact, not sampled;
- the low and high bounds a range estimate depends on are exact;
n_distinct and most_common_vals still need sampling, so this is a hybrid rather
than a replacement.
That is better statistics, not merely cheaper ones, and it is a genuine advantage of
columnar storage rather than a workaround.
Writing the results is supported on 18+, and not before
Checked on all three majors rather than assumed:
| function |
PG17 |
PG18 |
PG19 |
pg_restore_attribute_stats |
absent |
present |
present |
pg_restore_relation_stats |
absent |
present |
present |
pg_clear_attribute_stats |
absent |
present |
present |
So on 18 and later this writes statistics through a documented API. On 15 to 17 it
would mean writing pg_statistic directly, which is a materially different risk
(stakind slot layout, stavalues typing, staop/stacoll correctness) and a real
version-support decision rather than a detail. My inclination is 18+ only, and to
say so plainly rather than ship a fragile catalog writer for older majors.
The catch that should decide this
Autoanalyze will not call it. Same shape as pgcolumnar.vacuum(), but the
consequence differs in kind: a stale vacuum wastes space, while stale statistics
silently produce bad plans. We would be handing users a fast path that quietly rots,
and the failure mode is a query that got slower for no visible reason.
See #415, which proposes a background worker to schedule exactly this class of work.
If that lands, this becomes considerably more attractive, because the scheduling
gap is the main objection.
What I would do first, before building anything
Measure whether scan_analyze_next_tuple itself can be made competitive, because
that is the path autoanalyze actually takes. The slice-based group sampling is already
there; the open question is whether 622 ms against heap's 92 ms is inherent to decoding
whole groups for scattered sampled rows, or whether there is slack in it.
That measurement decides what this issue is. If the AM path can be brought close to
heap, a separate function is a maintenance burden with a stale-statistics footgun
attached. If it cannot, the function is the only route to statistics that match what
the storage can cheaply provide.
Proposed shape, if it goes ahead
pgcolumnar.analyze(rel regclass, columns text[] DEFAULT NULL)
Core ANALYZE stays the correctness path and keeps working with autoanalyze. This is
an opt-in accelerator for wide tables, documented as needing scheduling, exactly as
pgcolumnar.vacuum() is.
Found while answering how the table-AM interface's lack of projection support is
solved. See also #413, which is the same root cause on the index-build path and is
fixable, because that callback is told which columns it needs.
The problem
ANALYZEon a wide columnar table decodes every column, because the table-AM analyzecallbacks are never told which columns were asked for. Measured on 300,000 rows over 20
columns (one
int, nineteentextof 80 bytes), analysing one column:ANALYZE w (k)6.8x slower on the shape columnar storage should win.
The cheap fix does not work, and this is worth recording
The obvious repair is to make the analyze slot lazy, reusing the deferred-slot
machinery already used for index fetch, and let ANALYZE's own per-column access drive
the projection. It buys nothing.
acquire_sample_rowsdoes:It copies the whole tuple into a heap tuple and computes per-column statistics
afterwards, so every column is materialised at sample time regardless. Knowing the
column list would not help either: even
ANALYZE w (k)samples full tuples. That isa core design, not an oversight, and it is why an AM-side projection cannot fix this.
What a function could do that core cannot
Two things, and the second is the more interesting one.
1. Read one column at a time. Statistics are per attribute. A function can scan
each analysed column independently, which is exactly the access pattern the storage is
built for.
2. Take some statistics exactly, from metadata, without reading data at all. Zone
maps already hold per chunk
null_count,value_count, and min/max. So:null_fracis exact, not sampled;n_distinctandmost_common_valsstill need sampling, so this is a hybrid ratherthan a replacement.
That is better statistics, not merely cheaper ones, and it is a genuine advantage of
columnar storage rather than a workaround.
Writing the results is supported on 18+, and not before
Checked on all three majors rather than assumed:
pg_restore_attribute_statspg_restore_relation_statspg_clear_attribute_statsSo on 18 and later this writes statistics through a documented API. On 15 to 17 it
would mean writing
pg_statisticdirectly, which is a materially different risk(stakind slot layout,
stavaluestyping,staop/stacollcorrectness) and a realversion-support decision rather than a detail. My inclination is 18+ only, and to
say so plainly rather than ship a fragile catalog writer for older majors.
The catch that should decide this
Autoanalyze will not call it. Same shape as
pgcolumnar.vacuum(), but theconsequence differs in kind: a stale vacuum wastes space, while stale statistics
silently produce bad plans. We would be handing users a fast path that quietly rots,
and the failure mode is a query that got slower for no visible reason.
See #415, which proposes a background worker to schedule exactly this class of work.
If that lands, this becomes considerably more attractive, because the scheduling
gap is the main objection.
What I would do first, before building anything
Measure whether
scan_analyze_next_tupleitself can be made competitive, becausethat is the path autoanalyze actually takes. The slice-based group sampling is already
there; the open question is whether 622 ms against heap's 92 ms is inherent to decoding
whole groups for scattered sampled rows, or whether there is slack in it.
That measurement decides what this issue is. If the AM path can be brought close to
heap, a separate function is a maintenance burden with a stale-statistics footgun
attached. If it cannot, the function is the only route to statistics that match what
the storage can cheaply provide.
Proposed shape, if it goes ahead
Core
ANALYZEstays the correctness path and keeps working with autoanalyze. This isan opt-in accelerator for wide tables, documented as needing scheduling, exactly as
pgcolumnar.vacuum()is.Found while answering how the table-AM interface's lack of projection support is
solved. See also #413, which is the same root cause on the index-build path and is
fixable, because that callback is told which columns it needs.