What happens
A write in the same transaction before pgcolumnar.add_projection() makes every
later write in that transaction skip the new projection. The rows land in the base
table and never reach the projection, with no error. A covering projection scan and
pgcolumnar.read_projection() then answer as if those rows did not exist.
CREATE TABLE nt (a int, c int) USING pgcolumnar;
INSERT INTO nt SELECT g,g FROM generate_series(1,100) g;
BEGIN;
INSERT INTO nt SELECT g,g FROM generate_series(101,105) g; -- any write will do
SELECT pgcolumnar.add_projection('nt','np',ARRAY['a','c'],ARRAY['c']);
INSERT INTO nt SELECT g,g FROM generate_series(200,210) g; -- these 11 rows are lost to the projection
COMMIT;Measured on 8b39053 and on 8fb38ff, identically:
live base rows 116
rows in the projection 105 -- the 11 inserted after add_projection are missing
Control, the same transaction with the leading INSERT removed: 111 of 111, correct.
A leading DELETE is also fine, because DELETE does not write rows.
Why
PgColumnarProjectionFanoutRow builds the write state's projection-writer list on
first use and latches it, even when the list comes back empty
(src/columnar_write_state.c:3243-3266):
if (!baseWs->projInited)
{
... build baseWs->projWriters from PgColumnarListProjections(...) ...
baseWs->projInited = true;
}
if (baseWs->projWriters == NIL)
return;The write state lives for the transaction, keyed by (relid, subid). The first write
runs this with no projections declared, caches an empty list, and sets the latch.
add_projection then back-fills the rows that exist at that moment — which is why
the first 105 rows are present — but the later writes in the same transaction find
projInited already true and an empty projWriters, and return before writing
anything.
Scope, and a note on #873
The class is pre-existing, not introduced by #873. I checked before filing:
put an INSERT in front and both main and #873 fail identically, because INSERT
has always called the fan-out.
What #873 does is widen the trigger. Before it, an UPDATE never called the fan-out,
so BEGIN; UPDATE ...; add_projection(...); INSERT ...; COMMIT was correct on main
(111 of 111) and is wrong with #873 (100 of 111). That is not a reason to hold #873 —
it fixes silent wrong answers on a far more common path — but it is a second door
into this defect and it opens as soon as #873 lands.
What a fix has to do
Invalidate the latch when the set of projections changes, rather than trusting a
once-per-write-state cache. add_projection already knows it has changed the set;
the cheapest correct shapes are to flush or discard the relation's write state there,
or to make projInited a stamp compared against the projection catalog rather than a
boolean.
A regression arm belongs in test/projections.sh or test/projection_update.sh: the
transaction above, asserting the projection holds every live row, plus the
no-leading-write control so the arm cannot pass vacuously.
Provenance
Measured in container pgcolumnar-audit on PostgreSQL 19beta2 (/usr/local/pg19a),
base 8b39053 against 8fb38ff (#873), one tree per prefix, distinct .so
fingerprints per arm. Found while reviewing #873.
What happens
A write in the same transaction before
pgcolumnar.add_projection()makes everylater write in that transaction skip the new projection. The rows land in the base
table and never reach the projection, with no error. A covering projection scan and
pgcolumnar.read_projection()then answer as if those rows did not exist.Measured on
8b39053and on8fb38ff, identically:Control, the same transaction with the leading
INSERTremoved: 111 of 111, correct.A leading
DELETEis also fine, becauseDELETEdoes not write rows.Why
PgColumnarProjectionFanoutRowbuilds the write state's projection-writer list onfirst use and latches it, even when the list comes back empty
(
src/columnar_write_state.c:3243-3266):The write state lives for the transaction, keyed by (relid, subid). The first write
runs this with no projections declared, caches an empty list, and sets the latch.
add_projectionthen back-fills the rows that exist at that moment — which is whythe first 105 rows are present — but the later writes in the same transaction find
projInitedalready true and an emptyprojWriters, and return before writinganything.
Scope, and a note on #873
The class is pre-existing, not introduced by #873. I checked before filing:
put an
INSERTin front and bothmainand #873 fail identically, becauseINSERThas always called the fan-out.
What #873 does is widen the trigger. Before it, an
UPDATEnever called the fan-out,so
BEGIN; UPDATE ...; add_projection(...); INSERT ...; COMMITwas correct on main(111 of 111) and is wrong with #873 (100 of 111). That is not a reason to hold #873 —
it fixes silent wrong answers on a far more common path — but it is a second door
into this defect and it opens as soon as #873 lands.
What a fix has to do
Invalidate the latch when the set of projections changes, rather than trusting a
once-per-write-state cache.
add_projectionalready knows it has changed the set;the cheapest correct shapes are to flush or discard the relation's write state there,
or to make
projIniteda stamp compared against the projection catalog rather than aboolean.
A regression arm belongs in
test/projections.shortest/projection_update.sh: thetransaction above, asserting the projection holds every live row, plus the
no-leading-write control so the arm cannot pass vacuously.
Provenance
Measured in container
pgcolumnar-auditon PostgreSQL 19beta2 (/usr/local/pg19a),base
8b39053against8fb38ff(#873), one tree per prefix, distinct.sofingerprints per arm. Found while reviewing #873.