Skip to content

Report how much of a sorted table is still sorted (#301) - #312

Merged
ChronicallyJD merged 2 commits into
mainfrom
feat/301-sort-decay-visibility
Aug 1, 2026
Merged

Report how much of a sorted table is still sorted (#301)#312
ChronicallyJD merged 2 commits into
mainfrom
feat/301-sort-decay-visibility

Conversation

@ChronicallyJD

Copy link
Copy Markdown
Collaborator

Closes #301.

pgcolumnar.vacuum_sorted and pgcolumnar.cluster order a table once. Rows
inserted afterwards append in insertion order, so the sorted part shrinks in
proportion as the table grows. Nothing measured that, so the decision to re-sort
rested on a guess. This is option 1 from the issue: report it.

What it adds

pgcolumnar.sort_status(rel) returns one row:

Column Meaning
sort_key the declared sort_by key, or NULL
total_groups row groups in the table
sorted_groups row groups written by the last ordering rewrite
appended_groups row groups written after it
sorted_rows rows stored in the sorted groups
appended_rows rows stored in the appended groups

An ordering rewrite records the row group its run ends at, in a new
pgcolumnar.storage.sorted_through column.

Two design points that changed during the work

A boundary, not a count. The first version stored how many groups the sort
produced. That breaks: compact_rewrite retires a group inside the run and
writes its survivors back with a fresh, higher number, and those survivors are no
longer in the run's order. A count slides down onto the replacement and reports
the run intact. A boundary leaves it above the mark, where it belongs. There is a
removal proof for exactly this.

recluster does not set the mark. It reorders under
ShareUpdateExclusiveLock, so a group written by another session can take a
number inside its output range, and nothing distinguishes the two afterwards.
Recording a mark there would claim an order that may not hold. A table maintained
that way reports more decay than it has, which is the safe direction. Filed as
#311, and stated in the reference and in limitations.md.

Why the mark lives on the storage row

pgcolumnar.storage is keyed by storage id, and any rewrite creates a new
storage id. An unsorted pgcolumnar.vacuum therefore leaves the mark unset and
correctly reports the table as unsorted, with no invalidation step. A value in
pgcolumnar.options, which is keyed by relation, would outlive the layout it
describes.

Proofs

test/sort_status.sh, 45 checks, registered in the matrix. Five removal proofs,
each turning specific checks red:

Guard removed What went red
the mark in the plain-sort path 14 checks, including "a fresh sort reports every group sorted"
the mark in the Z-order path "cluster reports no decay", "cluster counts every row as sorted"
boundary replaced by a stored count "the run lost the retired groups", "the replacement counts as appended"
the mark set by an unsorted rewrite too the four unsorted-rewrite checks
a fresh storage row given an inherited mark 10 checks

One real defect surfaced while building it: heap_update rejected the storage
row as invisible, because the same command that flushed the rewrite's first group
had inserted it. Fixed with CommandCounterIncrement(), and the failure is what
proved the need for it.

Gates

Green on the five-major matrix (15, 16, 17, 18.4, 19beta2). test/docs_style.sh
passes.

Limits stated in the documentation

The counts describe where rows are stored, not whether their values are still in
order; an UPDATE stores the new version at the end, which counts as appended.
pgcolumnar.storage is not in pg_extension_config_dump, so a restored table
reports no sorted groups until it is sorted again.

jdatcmd and others added 2 commits July 31, 2026 18:52
pgcolumnar.vacuum_sorted and pgcolumnar.cluster order a table once. Rows
inserted afterwards append in insertion order, so the sorted part shrinks in
proportion as the table grows. Nothing measured that, so the decision to re-sort
rested on a guess.

An ordering rewrite now records the row group its run ends at, and
pgcolumnar.sort_status reports the split: sorted and appended groups and rows,
alongside the declared sort_by key.

The mark is a boundary, not a count. The online maintenance paths retire a group
and write its survivors back with a fresh, higher number; those survivors are no
longer in the run's order. A boundary leaves them above the mark, where they
belong. A count would slide down onto them as the run shrank and report an order
that is not there.

The mark lives on pgcolumnar.storage, keyed by storage id, because a storage row
has exactly the right lifetime. Any rewrite creates a new storage id, so an
unsorted vacuum leaves the mark unset and correctly reports the table as
unsorted, with no invalidation step. A value keyed by relation would outlive the
layout it describes.

The online recluster does not set the mark. It reorders under a lock that
permits concurrent inserts, so a group written by another session can take a
number inside its output range, and nothing distinguishes the two afterwards.
Recording a mark there would claim an order that may not hold. A table
maintained that way reports more decay than it has, which is the safe direction.
Filed as #311.

test/sort_status.sh, 45 checks. Each claim is proven by removal: dropping the
mark from the plain-sort path, from the Z-order path, storing a count instead of
a boundary, setting the mark from an unsorted rewrite, and giving a fresh
storage row an inherited mark each turn specific checks red.

Green on the five-major matrix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
pgcolumnar.storage is not in pg_extension_config_dump, so sorted_through is not
restored. A restored table therefore reports no sorted groups. The reference
claimed the function measures the order without stating that case.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
@ChronicallyJD
ChronicallyJD merged commit 43ffad7 into main Aug 1, 2026
11 checks passed
@ChronicallyJD
ChronicallyJD deleted the feat/301-sort-decay-visibility branch August 1, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sorted layout decays silently after inserts: no way to see skip quality has degraded

2 participants