Sorted layout decays silently after inserts: no way to see skip quality has degraded

[jdatcmd]

Raised while reviewing #292, and not a defect in it. The disclosure there is
correct; this is about what a user can do with it.

The gap

pgcolumnar.vacuum_sorted and pgcolumnar.cluster are one-shot physical
reorders. columnar_vacuum.c:754 states it plainly:

Sorting is a one-shot physical reorder; it is not persisted or
auto-maintained, so rows inserted afterward append in insert order.

That is the same contract as PostgreSQL CLUSTER, and it is a reasonable one.
The problem is not the decay. It is that the decay is invisible.

A sorted table gives tight, non-overlapping per-chunk minimum and maximum ranges,
so a range or equality filter on the sort key skips most chunk groups. Rows
appended afterwards land in insert order, and their groups span the whole key
range. Skipping degrades toward reading everything. Nothing tells the user this
has happened. The query still returns the right rows, and it quietly gets slower
until someone thinks to re-run the sort.

The measured effect in #292 is 2.6x to 3.0x on the queries a sort key is declared
for, and 116 of 667 chunk groups read against 1 of 667. That is the size of what
is silently lost.

Why this matters more now

Before #292 a sort was an explicit verb someone had just run, so its freshness was
in the operator's head. #292 makes the key declared and persistent: it is
recorded in pgcolumnar.options.sort_by and survives a dump. The declaration
outlives the memory of running it, and a user reasonably reads a persisted key as
a property of the table rather than as a note about something that happened once.

What is missing

A user cannot answer "is this table still sorted enough for the skipping to
work?" without measuring query behaviour. Three things could answer it, in
increasing order of cost:

1. Report it. pgcolumnar.stats already reports per-row-group row counts and
   sizes. The direct measure of skip quality is the overlap between row groups'
   zone-map ranges on the sort key: disjoint ranges skip, overlapping ranges do
   not. Adding that, or a simple "row groups written since the last sort", turns
   an invisible property into a query.

2. Estimate it from what already exists. ANALYZE collects correlation, and
   docs/features.md already says correlation is what lets the planner see the
   locality that vacuum_sorted and Z-ordering create. Correlation on the
   leading sort column falls as unsorted rows append, so it is a usable proxy
   with no new machinery.

3. Maintain it. Automatic or incremental re-sorting, or an online
   non-blocking re-sort for text keys. Declarative, persisted sort_by clustering key: text segment keys drive chunk-group skipping (#288) #292 lists both as follow-ups. This is the
   real fix and much the largest.

Recommendation

Option 1 before 1.0, and options 2 and 3 are separate work.

The argument is not performance, it is honesty about state. This project already
refuses to let a claim sit in the documentation without something that checks it,
and refuses a gate that reports success while running nothing. A declared sort key
whose value silently drains away, with no way to observe it, is the same shape:
the user believes a property holds because they declared it, and nothing tells
them it stopped holding.

Not urgent, and nothing is broken. A user who re-runs vacuum_sorted on a
schedule is fine today. This is about the user who declares a key, reads the
documentation correctly, and has no way to know when to act.