Skip to content

import_arrow and import_parquet do not maintain indexes: index scans silently return nothing, UNIQUE is not enforced #153

Description

@jdatcmd

Found while measuring the import path for a throughput plan. This is a
correctness defect, not a performance one.

Reproduction

PostgreSQL 17.10, 5,000 rows exported from one columnar table and imported into
another that was indexed before the import:

CREATE TABLE tgt (id int, v text) USING pgcolumnar;
CREATE INDEX tgt_id ON tgt (id);
SELECT pgcolumnar.import_arrow('tgt', '/tmp/f.arrows');   -- returns 5000
query result
count(*) via sequential scan 5000
count(*) WHERE id BETWEEN 100 AND 199, sequential scan 100
the same predicate via the index 0
count(*) WHERE id = 4242 via the index 0

The planner picks the index scan on its own:

Aggregate
  ->  Index Scan using tgt_id on tgt
        Index Cond: ((id >= 100) AND (id <= 199))

So the rows are in the table, absent from the index, and a perfectly ordinary
query returns nothing with no error.

A UNIQUE index is not enforced either. Importing the same 5,000 ids twice
into a table with CREATE UNIQUE INDEX gives 10,000 rows.

Both import_arrow and import_parquet behave identically. Verified on both.

Cause

The import loops call table_tuple_insert(rel, slot, cid, 0, NULL) per row.
That inserts into the table only. In PostgreSQL, index maintenance is the
executor's job (ExecInsertIndexTuples), not the access method's, so a function
that calls table_tuple_insert directly has to open the indexes and insert into
them itself. Nothing in columnar_arrow.c or columnar_parquet.c does.

The vacuum path already knows this: rewrite_one_group explicitly calls
rewrite_index_insert_row for exactly this reason, so the pattern to follow
exists in the tree.

Why no test caught it

No import suite creates an index on the target. arrow_import,
parquet_import, arrow_nested_import and parquet_nested_import all import
into a bare table and compare a sequential-scan result, which passes whether or
not the index exists.

Severity

Silent wrong answers on ordinary queries, and a UNIQUE constraint that does not
hold, on a documented, user-facing function. Same class as #137: nothing errors,
the data looks fine until something reads it through the index.

Fix

Two defensible options, and the choice is worth making deliberately:

  1. Maintain the indexes during import, as the rewrite path does. Correct, and
    keeps the function usable on a populated indexed table, but pays index cost
    per row and makes import slower still.
  2. Refuse to import into a table that has indexes, and document that indexes
    are created after loading. Cheap, honest, and matches the usual bulk-load
    pattern, but is a behaviour change for anyone relying on the current call
    succeeding.

Whichever is chosen, the import suites need a target with a plain index and a
target with a unique index, checking that an index scan and a sequential scan
agree and that a duplicate import raises rather than duplicating.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions