Found while measuring the import path for a throughput plan. This is a
correctness defect, not a performance one.
Reproduction
PostgreSQL 17.10, 5,000 rows exported from one columnar table and imported into
another that was indexed before the import:
CREATE TABLE tgt (id int, v text) USING pgcolumnar;
CREATE INDEX tgt_id ON tgt (id);
SELECT pgcolumnar.import_arrow('tgt', '/tmp/f.arrows'); -- returns 5000
| query |
result |
count(*) via sequential scan |
5000 |
count(*) WHERE id BETWEEN 100 AND 199, sequential scan |
100 |
| the same predicate via the index |
0 |
count(*) WHERE id = 4242 via the index |
0 |
The planner picks the index scan on its own:
Aggregate
-> Index Scan using tgt_id on tgt
Index Cond: ((id >= 100) AND (id <= 199))
So the rows are in the table, absent from the index, and a perfectly ordinary
query returns nothing with no error.
A UNIQUE index is not enforced either. Importing the same 5,000 ids twice
into a table with CREATE UNIQUE INDEX gives 10,000 rows.
Both import_arrow and import_parquet behave identically. Verified on both.
Cause
The import loops call table_tuple_insert(rel, slot, cid, 0, NULL) per row.
That inserts into the table only. In PostgreSQL, index maintenance is the
executor's job (ExecInsertIndexTuples), not the access method's, so a function
that calls table_tuple_insert directly has to open the indexes and insert into
them itself. Nothing in columnar_arrow.c or columnar_parquet.c does.
The vacuum path already knows this: rewrite_one_group explicitly calls
rewrite_index_insert_row for exactly this reason, so the pattern to follow
exists in the tree.
Why no test caught it
No import suite creates an index on the target. arrow_import,
parquet_import, arrow_nested_import and parquet_nested_import all import
into a bare table and compare a sequential-scan result, which passes whether or
not the index exists.
Severity
Silent wrong answers on ordinary queries, and a UNIQUE constraint that does not
hold, on a documented, user-facing function. Same class as #137: nothing errors,
the data looks fine until something reads it through the index.
Fix
Two defensible options, and the choice is worth making deliberately:
- Maintain the indexes during import, as the rewrite path does. Correct, and
keeps the function usable on a populated indexed table, but pays index cost
per row and makes import slower still.
- Refuse to import into a table that has indexes, and document that indexes
are created after loading. Cheap, honest, and matches the usual bulk-load
pattern, but is a behaviour change for anyone relying on the current call
succeeding.
Whichever is chosen, the import suites need a target with a plain index and a
target with a unique index, checking that an index scan and a sequential scan
agree and that a duplicate import raises rather than duplicating.
Found while measuring the import path for a throughput plan. This is a
correctness defect, not a performance one.
Reproduction
PostgreSQL 17.10, 5,000 rows exported from one columnar table and imported into
another that was indexed before the import:
count(*)via sequential scancount(*) WHERE id BETWEEN 100 AND 199, sequential scancount(*) WHERE id = 4242via the indexThe planner picks the index scan on its own:
So the rows are in the table, absent from the index, and a perfectly ordinary
query returns nothing with no error.
A UNIQUE index is not enforced either. Importing the same 5,000 ids twice
into a table with
CREATE UNIQUE INDEXgives 10,000 rows.Both
import_arrowandimport_parquetbehave identically. Verified on both.Cause
The import loops call
table_tuple_insert(rel, slot, cid, 0, NULL)per row.That inserts into the table only. In PostgreSQL, index maintenance is the
executor's job (
ExecInsertIndexTuples), not the access method's, so a functionthat calls
table_tuple_insertdirectly has to open the indexes and insert intothem itself. Nothing in
columnar_arrow.corcolumnar_parquet.cdoes.The vacuum path already knows this:
rewrite_one_groupexplicitly callsrewrite_index_insert_rowfor exactly this reason, so the pattern to followexists in the tree.
Why no test caught it
No import suite creates an index on the target.
arrow_import,parquet_import,arrow_nested_importandparquet_nested_importall importinto a bare table and compare a sequential-scan result, which passes whether or
not the index exists.
Severity
Silent wrong answers on ordinary queries, and a UNIQUE constraint that does not
hold, on a documented, user-facing function. Same class as #137: nothing errors,
the data looks fine until something reads it through the index.
Fix
Two defensible options, and the choice is worth making deliberately:
keeps the function usable on a populated indexed table, but pays index cost
per row and makes import slower still.
are created after loading. Cheap, honest, and matches the usual bulk-load
pattern, but is a behaviour change for anyone relying on the current call
succeeding.
Whichever is chosen, the import suites need a target with a plain index and a
target with a unique index, checking that an index scan and a sequential scan
agree and that a duplicate import raises rather than duplicating.