Coverage debt: two of the three visibility-map clears are held by nothing, and CHANGELOG says all three are

[OffgridwithJD]

Summary

CHANGELOG.md says of the three visibility-map clears added by #869: "All three clear now." Two of the three are held by nothing. Filed as coverage debt at jdatcmd's request, out of the #869 review. Not a defect — the code is right; the claim is unsupported by the tree.

Measurement

src/columnar_vacuum.c has three PgColumnarVMClearForRowRange call sites: recluster, the partial-group rewrite behind compact_rewrite, and expire. Delete the first two, keep expire's (source md5 47a19cd1a67f → ed1877bace13, call sites 3 → 1, mutation asserted applied by reading the mutated lines back out of the file), and every suite that could plausibly see it stays green on pg18a:

native_recluster          12/0    recluster_extent          17/0    recluster_gate      13/0
native_reclaim            33/0    native_reclaim_frag        4/0    native_reclaim_cycles 12/0
native_reclaim_reconcile  11/0    rewrite_group_scan         5/0    native_rewrite      17/0
native_ios                11/0    index_only                27/0    ttl_expire          34/0
vm_privilege              30/0    sorted_pathkeys          113/0

14 suites, 319 checks, zero failures with two of the three clears gone. Only expire's is covered, by test/ttl_expire.sh's index-only arm — which is the one the original data-loss report was about.

I tried to build the arm and it does not discriminate

Mirroring the ttl_ios idiom: VACUUM to set bits (relallvisible 2, plan confirmed an Index Only Scan), DELETE a quarter of the rows, recluster, then index-only against seqscan. Distinct .so per arm, mutation asserted applied:

                   branch (clear present)   mutated (clear removed)
.so                   182c9c9349c6               8a3f78663df6
index-only count          6000                       6000
seqscan count             6000                       6000
index-only sum(id)    24000000                   24000000
seqscan sum(id)       24000000                   24000000

Identical. The fixture cannot tell the two trees apart, so shipping it would have added an arm that cannot fail. I did not ship it.

State of knowledge, stated three ways

• Verified: the recluster and partial-group clears are uncovered.
• Verified: the fixture above cannot discriminate.
• Unknown: whether a fixture that can discriminate exists. I did not find one and I am not going to assert a mechanism I have not observed.

What blocks the honest arm

Reading the visibility map directly needs pg_visibility, which is not built under these prefixes (no pg_visibility.control in the extension directory). pg_class.relallvisible is not a substitute: it is a statistic VACUUM refreshes, and clearing a VM bit does not touch it — test/ttl_expire.sh records hitting exactly this wall and settles for asserting the clear through its consequence.

So the first concrete step is probably build contrib/pg_visibility alongside the test prefixes, which would let a suite assert the bits rather than a downstream effect, and would make expire's own arm stronger too.

Until then

Either the CHANGELOG sentence should say which of the three is actually held, or the arm should exist. Right now the sentence claims more than the tree can show, which is the thing this project does not merge — I am reporting my own branch's overclaim rather than leaving it for someone to find.

Measured on pg18a, tree #869 head 5ea1c95 (now merged as part of 4c024d0).

[OffgridwithJD]

Correction: the "unknown" in this issue has an answer, and it was a defect in my fixture

I filed this saying: verified the two clears are uncovered; verified my candidate fixture cannot discriminate; unknown whether a discriminating fixture exists. The third of those was wrong to leave open. jdatcmd named the mechanism on #878 and I measured it.

My fixture deleted a quarter of the rows spread across every block. DELETE FROM vmr WHERE id % 4 = 0. A DELETE clears the all-visible bits of the blocks holding the deleted rows, so that delete cleared all of them:

                                                 interior blocks all-visible
after VACUUM                                             69 / 69
after my DELETE (id % 4 = 0)                              0 / 69
after a front-only DELETE                                67 / 69

With no all-visible bit anywhere, there is no stale bit for the missing clear to leave behind, and no block for an index-only scan to wrongly trust. The arm could not fail whatever the code did. That is a nameable defect in the fixture, not evidence that no fixture exists.

I also want to correct something I implied. I cited pg_class.relallvisible being a VACUUM statistic as the wall — that is real, and test/ttl_expire.sh records hitting it, but my own fixture never read relallvisible; it compared an index-only scan against a seqscan. So the relallvisible wall explains why ttl_expire.sh asserts through a consequence. It does not explain why my fixture was dead. Two different problems, and I ran them together.

And the instrument I said was missing was already in the tree. I wrote that this "wants pg_visibility built under these prefixes". It does not: pgcolumnar.vm_is_visible(rel, blk) reads the fork through visibilitymap_get_status, the same call the index-only-scan executor makes. It has been there since before 1.0-alpha2 — it is in both frozen base fixtures — and neither of us reached for it. The suggested first step in this issue was wrong.

Status

The coverage half is closed by #878, which adds test/vm_clear_on_renumber.sh: :346 neutered gives 16+1, :765 neutered gives 15+2, and I reproduced both independently. Once that merges, the All three clear now sentence this issue was filed against is true of the tree rather than only of the code.

Leaving this open until #878 lands.