Report how much of a sorted table is still sorted (#301) - #312
Merged
Conversation
pgcolumnar.vacuum_sorted and pgcolumnar.cluster order a table once. Rows inserted afterwards append in insertion order, so the sorted part shrinks in proportion as the table grows. Nothing measured that, so the decision to re-sort rested on a guess. An ordering rewrite now records the row group its run ends at, and pgcolumnar.sort_status reports the split: sorted and appended groups and rows, alongside the declared sort_by key. The mark is a boundary, not a count. The online maintenance paths retire a group and write its survivors back with a fresh, higher number; those survivors are no longer in the run's order. A boundary leaves them above the mark, where they belong. A count would slide down onto them as the run shrank and report an order that is not there. The mark lives on pgcolumnar.storage, keyed by storage id, because a storage row has exactly the right lifetime. Any rewrite creates a new storage id, so an unsorted vacuum leaves the mark unset and correctly reports the table as unsorted, with no invalidation step. A value keyed by relation would outlive the layout it describes. The online recluster does not set the mark. It reorders under a lock that permits concurrent inserts, so a group written by another session can take a number inside its output range, and nothing distinguishes the two afterwards. Recording a mark there would claim an order that may not hold. A table maintained that way reports more decay than it has, which is the safe direction. Filed as #311. test/sort_status.sh, 45 checks. Each claim is proven by removal: dropping the mark from the plain-sort path, from the Z-order path, storing a count instead of a boundary, setting the mark from an unsorted rewrite, and giving a fresh storage row an inherited mark each turn specific checks red. Green on the five-major matrix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
pgcolumnar.storage is not in pg_extension_config_dump, so sorted_through is not restored. A restored table therefore reports no sorted groups. The reference claimed the function measures the order without stating that case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #301.
pgcolumnar.vacuum_sortedandpgcolumnar.clusterorder a table once. Rowsinserted afterwards append in insertion order, so the sorted part shrinks in
proportion as the table grows. Nothing measured that, so the decision to re-sort
rested on a guess. This is option 1 from the issue: report it.
What it adds
pgcolumnar.sort_status(rel)returns one row:sort_keysort_bykey, or NULLtotal_groupssorted_groupsappended_groupssorted_rowsappended_rowsAn ordering rewrite records the row group its run ends at, in a new
pgcolumnar.storage.sorted_throughcolumn.Two design points that changed during the work
A boundary, not a count. The first version stored how many groups the sort
produced. That breaks:
compact_rewriteretires a group inside the run andwrites its survivors back with a fresh, higher number, and those survivors are no
longer in the run's order. A count slides down onto the replacement and reports
the run intact. A boundary leaves it above the mark, where it belongs. There is a
removal proof for exactly this.
reclusterdoes not set the mark. It reorders underShareUpdateExclusiveLock, so a group written by another session can take anumber inside its output range, and nothing distinguishes the two afterwards.
Recording a mark there would claim an order that may not hold. A table maintained
that way reports more decay than it has, which is the safe direction. Filed as
#311, and stated in the reference and in
limitations.md.Why the mark lives on the storage row
pgcolumnar.storageis keyed by storage id, and any rewrite creates a newstorage id. An unsorted
pgcolumnar.vacuumtherefore leaves the mark unset andcorrectly reports the table as unsorted, with no invalidation step. A value in
pgcolumnar.options, which is keyed by relation, would outlive the layout itdescribes.
Proofs
test/sort_status.sh, 45 checks, registered in the matrix. Five removal proofs,each turning specific checks red:
One real defect surfaced while building it:
heap_updaterejected the storagerow as invisible, because the same command that flushed the rewrite's first group
had inserted it. Fixed with
CommandCounterIncrement(), and the failure is whatproved the need for it.
Gates
Green on the five-major matrix (15, 16, 17, 18.4, 19beta2).
test/docs_style.shpasses.
Limits stated in the documentation
The counts describe where rows are stored, not whether their values are still in
order; an
UPDATEstores the new version at the end, which counts as appended.pgcolumnar.storageis not inpg_extension_config_dump, so a restored tablereports no sorted groups until it is sorted again.