From 6488fb3e36b9e61a58ab1c0ce38686f824003761 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 24 Jul 2026 11:42:34 -0600 Subject: [PATCH 1/2] docs: document the external-Parquet read surface, correct stale claims Phase G added reading external Parquet in place (read_parquet, parquet_schema, the pgcolumnar_parquet FDW, directory/glob paths, predicate and projection pushdown, more codecs, and uuid/numeric/time-unit read coverage), and the docs were never updated. Some claims had become false. Corrections: - limitations.md marked uuid and numeric Parquet import as unsupported; both are supported now. Only json/jsonb import remains unsupported. - features.md and sql-reference.md said the reader decompresses Snappy only; it now also reads GZIP, ZSTD, and LZ4_RAW. Additions: - sql-reference.md: read_parquet, parquet_schema, and the FDW, plus directory and glob paths on import_parquet. - features.md: an external-Parquet section. - user-guide.md: a worked example with an EXPLAIN showing row-group and column skipping. - limitations.md: read codecs, and the read-surface limits (no partition pruning, no recursive walk, no streaming, nanosecond timestamps advised as bigint, INT32/INT64 DECIMAL not read). - ARCHITECTURE.md: the shared scan core, the three surfaces, pushdown, multi-file, and the crafted-file hardening. - testing.md: the new suites. - CHANGELOG.md: the read surface, extended type coverage, and the reader hardening. - ROADMAP.md: Phase G read surface marked complete; future directions narrowed to ORC, open table formats, and the Parquet follow-ons still open. design/PHASE_G_DOCS_AUDIT.md records the audit. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 25 ++++++++++++--- design/PHASE_G_DOCS_AUDIT.md | 62 ++++++++++++++++++++++++++++++++++++ design/ROADMAP.md | 15 +++++++-- docs/ARCHITECTURE.md | 40 ++++++++++++++++------- docs/features.md | 22 +++++++++++-- docs/limitations.md | 45 +++++++++++++++++++++----- docs/sql-reference.md | 61 +++++++++++++++++++++++++++++++++-- docs/testing.md | 10 ++++++ docs/user-guide.md | 36 +++++++++++++++++++++ 9 files changed, 286 insertions(+), 30 deletions(-) create mode 100644 design/PHASE_G_DOCS_AUDIT.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 39f5dd7..6db5ca8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,10 +40,21 @@ unreleased. For the forward-looking plan see one-dimensional arrays, and composite types, with nulls at every level. - Arrow IPC and Parquet import (`pgcolumnar.import_arrow`, `pgcolumnar.import_parquet`). The Parquet reader parses Thrift metadata, - decompresses Snappy, and decodes PLAIN and dictionary encodings from data-page - versions 1 and 2. Both readers reconstruct one-dimensional arrays and composite - types: Arrow from its List and Struct buffers, Parquet from the Dremel - repetition and definition levels. + decompresses uncompressed, Snappy, GZIP, ZSTD, and LZ4_RAW pages, and decodes + PLAIN and dictionary encodings from data-page versions 1 and 2. Both readers + reconstruct one-dimensional arrays and composite types: Arrow from its List and + Struct buffers, Parquet from the Dremel repetition and definition levels. +- Reading external Parquet in place. `pgcolumnar.read_parquet(path)` returns a + file's rows without importing, `pgcolumnar.parquet_schema(path)` reports its + columns and inferred types, and the `pgcolumnar_parquet` foreign-data wrapper + exposes a file as a foreign table. A `path` may be a single file, a directory + of `*.parquet` files, or a glob pattern, read as one relation in sorted order. + The foreign scan skips row groups excluded by the query's predicate (min/max + statistics) and decodes only the referenced columns; `EXPLAIN ANALYZE` reports + the row groups and columns read and skipped and the number of files. +- Parquet read type coverage extended to uuid and numeric (from fixed and + variable DECIMAL, precision up to 38), fixed-length binary, and millisecond, + microsecond, and nanosecond time units. - User and administrator documentation under [docs/](docs/index.md): installation, user guide, administration, configuration reference, SQL reference, and limitations. @@ -60,6 +71,12 @@ unreleased. For the forward-looking plan see them, using memory proportional to the row count. They now reset a per-row scratch context (and, for Parquet, a per-row-group context for decoded leaf streams), so peak memory stays bounded on large files. +- Hardened the Parquet reader against crafted files. File-declared page sizes, + DECIMAL scale, and per-row-group column-chunk counts are range-checked, so a + malformed footer yields a clean decode error rather than a stack overflow, an + out-of-bounds read, or a wrong value. Float and double row-group skipping + accounts for NaN and for inverted min/max intervals, and narrowing a wide + Parquet value into a smaller PostgreSQL type raises instead of wrapping. - Concurrent inserts of the same unique-index key now serialize correctly with a transaction-scoped advisory lock (`pgcolumnar.enable_unique_insert_lock`). - Lost delete marks under concurrent same-chunk-group deletes. diff --git a/design/PHASE_G_DOCS_AUDIT.md b/design/PHASE_G_DOCS_AUDIT.md new file mode 100644 index 0000000..2409f1d --- /dev/null +++ b/design/PHASE_G_DOCS_AUDIT.md @@ -0,0 +1,62 @@ +# Phase G documentation audit + +Reconciles the user and engineering docs against the external-Parquet read +surface, which landed across #98-#101 and the follow-ons #106/#107/#109/#111 and +was never documented. Some existing claims are now false. Style per project +convention: professional, no em-dashes, no unnecessary adjectives. + +## What is now true but undocumented or mis-documented + +Read surfaces, all superuser, server-side paths: +- `pgcolumnar.read_parquet(path) AS t(...)` -- SRF, reads rows in place. +- `pgcolumnar.parquet_schema(path)` -- reports leaf columns and the PostgreSQL + type each maps to. +- The `pgcolumnar_parquet` foreign-data wrapper -- `CREATE SERVER` + + `CREATE FOREIGN TABLE ... OPTIONS (path ...)`. +- Predicate pushdown (row-group skipping via min/max stats) and column projection + pushdown, both visible in `EXPLAIN ANALYZE` (Row Groups, Row Groups Skipped, + Columns Read/Total). +- A `path` that is a directory reads every `*.parquet` in it; a glob pattern + expands; sorted, deterministic. +- Codecs on read: uncompressed, Snappy, GZIP, ZSTD, LZ4_RAW. +- FLBA types on read: uuid and numeric(p,s) via DECIMAL, plus fixed bytea. + +## Corrections (currently wrong) + +- `docs/limitations.md` type matrix: `uuid` and `numeric` Parquet import are + marked "no" -- both are "yes" now (#106). `json`/`jsonb` import stays "no". +- `docs/limitations.md` prose "does not currently import those three types from + Parquet" -- now only json. +- `docs/features.md` "decompresses Snappy" -- now uncompressed, Snappy, GZIP, + ZSTD, LZ4_RAW. +- `docs/sql-reference.md` import_parquet "handles Snappy compression" -- same + codec list update. + +## Additions + +- `docs/sql-reference.md`: new sections for `read_parquet`, `parquet_schema`, and + the FDW; note directory/glob paths on all read paths; note pushdown in EXPLAIN. +- `docs/features.md`: an "external Parquet" bullet group -- read in place via SRF + or FDW, directory/glob, predicate and projection pushdown, codec list. +- `docs/user-guide.md`: a worked example of read_parquet and a foreign table over + a directory, with an EXPLAIN showing skipping. +- `docs/ARCHITECTURE.md`: the shared scan core, the two surfaces, and where + pushdown sits. +- `CHANGELOG.md`: entries for the read surface and the follow-ons. +- `design/ROADMAP.md`: mark the Parquet read follow-ons done. + +## Still-true limitations to state + +- No json/jsonb import from Parquet. +- INT32/INT64-backed DECIMAL not read (only FLBA/BYTE_ARRAY DECIMAL); the schema + does not advertise numeric for them. +- TIMESTAMP_NANOS advises `bigint` (lossless); declaring `timestamp` truncates. +- No Hive-style partition pruning, no recursive directory walk, no streaming + (each file is read fully into memory), single schema per directory assumed by + `parquet_schema`. +- LZO, BROTLI, and the deprecated Hadoop-framed LZ4 (codec 5) are not read. +- Reads are superuser-only and little-endian only, as export/import already are. + +## Order of work +limitations (fix false claims first) -> features -> sql-reference -> user-guide +-> ARCHITECTURE -> CHANGELOG -> ROADMAP. diff --git a/design/ROADMAP.md b/design/ROADMAP.md index caf219b..d9190ea 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -29,6 +29,13 @@ matrix. Gap specifications are in [gaps/](gaps/). | Arrow nested import (List → array, Struct → composite) | gap 27 | | Parquet nested import (LIST → array, group → composite; Dremel level assembly) | gap 27 | | **Gap 27 complete** — Arrow/Parquet interop: export + import, flat + nested, both formats | gap 27 | +| Read external Parquet in place: `read_parquet`, `parquet_schema`, `pgcolumnar_parquet` FDW | Phase G | +| Parquet FDW predicate pushdown (row-group skipping) and column projection pushdown | Phase G | +| Parquet read codecs: GZIP, ZSTD, LZ4_RAW (added to uncompressed and Snappy) | Phase G | +| Parquet read type coverage: uuid, numeric via DECIMAL, fixed binary, ms/us/ns time units | Phase G | +| Multi-file reads: a directory of `*.parquet` or a glob read as one relation | Phase G | +| Reader hardening against crafted files (scale/size/chunk-count guards, NaN and inverted stats) | Phase G | +| **Phase G read surface complete** — external Parquet read, pushdown, multi-file, all matrix-gated | Phase G | ## Remaining @@ -157,9 +164,11 @@ compression defaults, and the FastLanes on-disk format generation. The research pass returned few surviving primary sources in this area, so these are directions to investigate and spec, not validated recommendations: -- Query external Parquet, ORC, and Arrow files with predicate and projection - pushdown, and read open table formats (Apache Iceberg, Delta Lake, Hudi), - reusing the zone-map and delete-vector machinery for file pruning. +- External Parquet read with predicate and projection pushdown is done (Phase G, + see above). What remains here: ORC, open table formats (Apache Iceberg, Delta + Lake, Hudi), and, within Parquet, Hive-style partition pruning, recursive + directory walks, streaming instead of reading each file fully into memory, and + INT32/INT64-backed DECIMAL reads. - Arrow C Data Interface zero-copy export, and Arrow Flight SQL or ADBC access. - New PostgreSQL 17-19 integration points: read stream and asynchronous IO (partly used), `MERGE`, incremental materialized views (pg_ivm), logical decoding of diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ea8a63c..b7c952d 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -222,17 +222,35 @@ levels (Dremel): scalars (one leaf), 1-D arrays (a 3-level LIST), and composites dependency; same scalar type coverage as the Arrow writer. ### columnar_parquet_reader.c -Parquet import (`pgcolumnar.import_parquet`, gap 27). A self-contained reader: a -Thrift compact-protocol decoder for the footer and page headers, clean-room -Snappy decompression, the RLE/bit-packed hybrid decoder for repetition/definition -levels and dictionary indices, PLAIN and dictionary value decoding, and both -DATA_PAGE v1 and v2 (what pyarrow writes). The schema tree is walked to derive -each leaf column's Dremel level bounds; nested columns are reconstructed by -decoding each leaf's full entry sequence (defs/reps/dense values) and grouping -repeated runs (LIST to array, group to composite), the inverse of the nested -Parquet exporter. Scalars remain byte-for-byte the flat path. Rows are inserted -into an existing target table via `table_tuple_insert`, mirroring the Arrow -importer. No libparquet dependency. +Parquet import and the external-Parquet read surface (gap 27 and Phase G). A +self-contained reader: a Thrift compact-protocol decoder for the footer and page +headers, clean-room Snappy plus GZIP (zlib), ZSTD, and LZ4_RAW page +decompression, the RLE/bit-packed hybrid decoder for repetition/definition levels +and dictionary indices, PLAIN and dictionary value decoding, and both DATA_PAGE +v1 and v2 (what pyarrow writes). The schema tree is walked to derive each leaf +column's Dremel level bounds; nested columns are reconstructed by decoding each +leaf's full entry sequence (defs/reps/dense values) and grouping repeated runs +(LIST to array, group to composite), the inverse of the nested Parquet exporter. +Scalars remain byte-for-byte the flat path. No libparquet dependency. + +One shared row-producing core (`pq_read_rows`, decode one row group into slots) +feeds three surfaces over the same file parse and type inference: + +- `import_parquet` inserts into a target table via `table_tuple_insert`. +- `read_parquet` returns rows as a set-returning function, and `parquet_schema` + reports the leaf columns and their inferred PostgreSQL types. +- The `pgcolumnar_parquet` foreign-data wrapper materializes the file into a + tuplestore drained by the scan. It pushes down predicates by skipping row + groups whose min/max statistics prove empty (only fixed-width ordered types, + with NaN and inverted-interval guards), and projects by decoding only the + columns the plan references (computed from the base rel's reltarget and quals). + +A `path` that is a directory or glob resolves to a sorted list of files, each +read through the same core into the one sink; per-file decode buffers are freed +between files. Decode paths are hardened against crafted files: file-declared +sizes, DECIMAL scale, and per-row-group chunk counts are all range-checked so a +malformed footer yields a clean error rather than an out-of-bounds read or a +wrong value. ### columnar_visibilitymap.c Index-only-scan support (gap 28). A columnar visibility-map fork records which diff --git a/docs/features.md b/docs/features.md index 0cff194..ca8caa2 100644 --- a/docs/features.md +++ b/docs/features.md @@ -119,10 +119,28 @@ coverage. dependency. - Import from Arrow and Parquet: `pgcolumnar.import_arrow(table, path)` and `pgcolumnar.import_parquet(table, path)` into an existing target table. The - Parquet reader parses Thrift metadata, decompresses Snappy, and decodes PLAIN - and dictionary encodings from data-page versions 1 and 2. + Parquet reader parses Thrift metadata, decompresses uncompressed, Snappy, GZIP, + ZSTD, and LZ4_RAW pages, and decodes PLAIN and dictionary encodings from + data-page versions 1 and 2. - Both directions cover scalar types, one-dimensional arrays, and composite types (Arrow List and Struct, Parquet LIST and group), with nulls at every level. The functions require superuser and run on little-endian hosts. See the [SQL reference](sql-reference.md#import-and-export) and the [type-coverage table](limitations.md#import-and-export-type-coverage). + +## Reading external Parquet in place + +- `pgcolumnar.read_parquet(path) AS t(...)` reads a server-side Parquet file's + rows without importing them, and `pgcolumnar.parquet_schema(path)` reports its + leaf columns and the PostgreSQL type each maps to. +- The `pgcolumnar_parquet` foreign-data wrapper exposes a Parquet file as a + foreign table: `CREATE FOREIGN TABLE ... SERVER ... OPTIONS (path '...')`. +- A `path` that is a directory reads every `*.parquet` file inside it as one + relation, and a glob pattern expands the same way, in a deterministic sorted + order. +- The foreign-table scan pushes work down: row groups whose min/max statistics + exclude the query's predicate are skipped, and only the columns the query + references are decoded. `EXPLAIN ANALYZE` reports the row groups read and + skipped, the columns read, and the number of files. +- uuid and numeric columns are read from their Parquet representations, and the + reader handles millisecond, microsecond, and nanosecond time units. diff --git a/docs/limitations.md b/docs/limitations.md index a9ab38f..e2032d2 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -139,18 +139,47 @@ listed are rejected. | `text`, `varchar` | yes | yes | yes | yes | | `bytea` | yes | yes | yes | yes | | `date`, `time`, `timestamp`, `timestamptz` | yes | yes | yes | yes | -| `uuid` | yes | yes | yes | no | -| `numeric` | yes | yes | yes | no | +| `uuid` | yes | yes | yes | yes | +| `numeric` | yes | yes | yes | yes | | `json`, `jsonb` | yes | yes | yes | no | | one-dimensional array of the above | yes | yes | yes | yes | | composite of the above | yes | yes | yes | yes | -`uuid`, `numeric`, and `json` can be exported to Parquet and read back with other -tools, but pgColumnar does not currently import those three types from Parquet. -They are supported end to end through Arrow. +`uuid` is imported from a 16-byte fixed-length binary column, and `numeric` from +a DECIMAL column stored as fixed or variable big-endian bytes with precision up +to 38. A DECIMAL backed by an INT32 or INT64 physical column is not yet read. +`json` and `jsonb` can be exported to Parquet and read back with other tools, but +pgColumnar does not currently import them; they are supported end to end through +Arrow. ## Compression codecs -`lz4` and `zstd` are available only when the extension was built with the -corresponding system libraries. When a codec is not built in, a request for it -falls back to a codec that is present. `pglz` and `none` are always available. +For the native table format, `lz4` and `zstd` are available only when the +extension was built with the corresponding system libraries. When a codec is not +built in, a request for it falls back to a codec that is present. `pglz` and +`none` are always available. + +When reading external Parquet files, the reader decodes uncompressed, Snappy, +GZIP, ZSTD, and LZ4_RAW pages. GZIP requires a build with zlib, and ZSTD and +LZ4_RAW require the same libraries as the native codecs; a page whose codec was +not built in fails with a clean decode error. LZO, BROTLI, and the deprecated +Hadoop-framed LZ4 (codec 5, as distinct from LZ4_RAW) are not read. + +## Reading external Parquet + +The read-in-place surface (`read_parquet`, `parquet_schema`, and the +`pgcolumnar_parquet` foreign-data wrapper) has these limits: + +- Reads are superuser only and run on little-endian hosts, as import and export + do, since they read a server-side path. +- A `path` that is a directory reads the `*.parquet` files directly inside it; + there is no recursive walk and no Hive-style partition pruning (directory names + of the form `col=value` are not exposed as columns). +- `parquet_schema` describes the first file of a directory or glob, assuming the + set is uniform. The read paths still bind every file against the declared + columns, so a mismatched file raises rather than returning wrong rows. +- A `TIMESTAMP` column with nanosecond precision is advised as `bigint`, which is + exact; declaring it `timestamp` reads it with the sub-microsecond digits + truncated. +- Each file is read fully into memory before its rows are produced; there is no + streaming. diff --git a/docs/sql-reference.md b/docs/sql-reference.md index d3af37d..315d193 100644 --- a/docs/sql-reference.md +++ b/docs/sql-reference.md @@ -150,14 +150,71 @@ rows inserted. ### pgcolumnar.import_parquet(rel regclass, path text) returns bigint Inserts the rows of a Parquet file at `path` into the existing table `rel`. The -reader handles Snappy compression, PLAIN and dictionary encodings, and data-page -versions 1 and 2. Returns the number of rows inserted. +reader handles uncompressed, Snappy, GZIP, ZSTD, and LZ4_RAW pages, PLAIN and +dictionary encodings, and data-page versions 1 and 2. `path` may name a single +file, a directory (every `*.parquet` file inside it is imported), or a glob +pattern. Returns the number of rows inserted. ```sql -- round-trip a table through Parquet SELECT pgcolumnar.export_parquet('events', '/tmp/events.parquet'); -- returns row count CREATE TABLE events_copy (LIKE events) USING pgcolumnar; SELECT pgcolumnar.import_parquet('events_copy', '/tmp/events.parquet'); + +-- import an entire directory of Parquet files +SELECT pgcolumnar.import_parquet('events_copy', '/data/events/'); +``` + +## Reading external Parquet + +These read a server-side Parquet file in place, without importing it. They +require superuser and run on little-endian hosts. In every case `path` may be a +single file, a directory (all `*.parquet` files inside it, read as one relation +in sorted order), or a glob pattern. + +### pgcolumnar.read_parquet(path text) returns setof record + +Returns the rows of a Parquet file. The caller supplies a column definition list +that names the output columns and their types; the reader binds it against the +file's leaf columns by position, with the same type-compatibility rules as +import. + +```sql +SELECT * FROM pgcolumnar.read_parquet('/data/events.parquet') + AS t(id int, ts timestamp, amount numeric(12,2)); + +-- read a whole directory +SELECT count(*) FROM pgcolumnar.read_parquet('/data/events/') + AS t(id int, ts timestamp, amount numeric(12,2)); +``` + +### pgcolumnar.parquet_schema(path text) returns table(column_name text, data_type text, nullable bool) + +Reports the leaf columns of a Parquet file and the PostgreSQL type each maps to, +without reading the data. Useful for writing the column definition list for +`read_parquet` or a foreign table. For a directory or glob it describes the first +file. + +```sql +SELECT * FROM pgcolumnar.parquet_schema('/data/events.parquet'); +``` + +### The pgcolumnar_parquet foreign-data wrapper + +Exposes a Parquet file, directory, or glob as a foreign table. The scan pushes +work down: row groups whose min/max statistics exclude the query's predicate are +skipped, and only referenced columns are decoded. + +```sql +CREATE SERVER pq FOREIGN DATA WRAPPER pgcolumnar_parquet; +CREATE FOREIGN TABLE events (id int, ts timestamp, amount numeric(12,2)) + SERVER pq OPTIONS (path '/data/events/'); + +SELECT sum(amount) FROM events WHERE ts >= '2026-01-01'; + +-- EXPLAIN ANALYZE reports Row Groups, Row Groups Skipped, Columns Read, +-- Columns Total, and Files. +EXPLAIN (ANALYZE, COSTS OFF) SELECT id FROM events WHERE ts >= '2026-01-01'; ``` ## Visibility map inspection diff --git a/docs/testing.md b/docs/testing.md index db94832..97b0aa4 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -31,6 +31,16 @@ test/arrow_nested.sh /path/to/pg_config # nested Arrow export test/parquet_nested.sh /path/to/pg_config # nested Parquet export test/arrow_nested_import.sh /path/to/pg_config # nested Arrow import test/parquet_nested_import.sh /path/to/pg_config # nested Parquet import +test/native_parquet_schema.sh /path/to/pg_config # parquet_schema type inference +test/native_read_parquet.sh /path/to/pg_config # read_parquet in place +test/native_parquet_fdw.sh /path/to/pg_config # pgcolumnar_parquet foreign tables +test/native_parquet_pushdown.sh /path/to/pg_config # FDW row-group predicate skipping +test/native_parquet_projection.sh /path/to/pg_config # FDW column projection pushdown +test/native_parquet_units.sh /path/to/pg_config # TIME/TIMESTAMP unit handling +test/native_parquet_flba.sh /path/to/pg_config # uuid, decimal, fixed binary reads +test/native_parquet_codecs.sh /path/to/pg_config # GZIP, ZSTD, LZ4_RAW page reads +test/native_parquet_hardening.sh /path/to/pg_config # crafted-file decode guards +test/native_parquet_multifile.sh /path/to/pg_config # directory and glob reads test/native_writer.sh /path/to/pg_config # native format catalog output test/native_roundtrip.sh /path/to/pg_config # native write then read round-trip test/native_encoding.sh /path/to/pg_config # native per-vector encoding cascade diff --git a/docs/user-guide.md b/docs/user-guide.md index afcbb30..d309567 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -126,6 +126,42 @@ CREATE TABLE people ( INSERT INTO people VALUES (1, ARRAY['a','b'], ROW('Portland','97201')::addr); ``` +## Read Parquet files in place + +You can query a server-side Parquet file without importing it, either as a +set-returning function or as a foreign table. Both require superuser. + +```sql +-- inspect the file's columns and their inferred types +SELECT * FROM pgcolumnar.parquet_schema('/data/events.parquet'); + +-- read the rows directly +SELECT count(*), max(ts) +FROM pgcolumnar.read_parquet('/data/events.parquet') + AS t(id int, ts timestamp, amount numeric(12,2)); +``` + +For repeated queries, a foreign table is more convenient, and its scans skip row +groups and columns the query does not need: + +```sql +CREATE SERVER pq FOREIGN DATA WRAPPER pgcolumnar_parquet; +CREATE FOREIGN TABLE events (id int, ts timestamp, amount numeric(12,2)) + SERVER pq OPTIONS (path '/data/events/'); -- a directory of *.parquet files + +SELECT sum(amount) FROM events WHERE ts >= '2026-01-01'; +``` + +A `path` may name one file, a directory (all `*.parquet` files inside it), or a +glob pattern. `EXPLAIN (ANALYZE)` shows how much the scan skipped: + +```sql +EXPLAIN (ANALYZE, COSTS OFF) SELECT id FROM events WHERE ts >= '2026-01-01'; +-- Foreign Scan on events +-- Row Groups: 12 Row Groups Skipped: 9 +-- Columns Read: 2 Columns Total: 3 Files: 4 +``` + ## Next steps - Operate tables in production: [Administration](administration.md). From d5d28fd522a1e4605e3221dba4eb6a40f9da4248 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 24 Jul 2026 11:55:24 -0600 Subject: [PATCH 2/2] docs: address the #112 review, and correct the in-database Parquet comments Review findings from #112, each verified against the code before the change: - numeric Parquet round trip is qualified. The exporter writes DECIMAL only for numeric(p,s) with p <= 38 (columnar_parquet.c:364-385); anything else, including an unconstrained numeric, becomes UTF8, and the import binder has no NUMERICOID case for a text leaf, so the round trip fails. The type table now says "yes (numeric(p,s), p<=38)" for export and "yes (DECIMAL only)" for import, with a paragraph telling a user to declare a precision. Arrow carries either form (it has the same text fallback and imports through the type input function). - installation.md gets the pass the other pages got: zlib listed as an optional dependency gating GZIP Parquet pages; the codec-fallback bullet scoped to the native format, since a Parquet page in a codec that was not built in fails rather than falling back; the little-endian requirement extended to the read surface; and the version range corrected to 15 through 19 to match README.md, docs/index.md, and the matrix. - The column definition list must cover every leaf column. build_imp_targets errors unless it consumes exactly pf->ncols, so a subset is an error and not a projection. Stated in sql-reference.md and limitations.md, next to the projection text so declaring and decoding are not confused. - Row-group skipping preconditions are documented. Var op Const only (no Param, which is what lets the skip set be computed once), INT32/INT64/FLOAT/DOUBLE physical types only, exact constant type, both statistics present and not inverted. features.md and sql-reference.md now point at that list rather than promising unconditional skipping. - The in-database comments still described a single file. import_parquet, parquet_schema, read_parquet, and the foreign-data wrapper (the text \dew+ prints) now say file, directory, or glob, which is what #111 made true. Nits: the audit doc said two surfaces where the change says three; the user-guide EXPLAIN sample now puts one property per line, as ExplainPropertyInteger emits them. Co-Authored-By: Claude Opus 5 (1M context) --- design/PHASE_G_DOCS_AUDIT.md | 2 +- docs/features.md | 4 +++- docs/installation.md | 17 +++++++++++++---- docs/limitations.md | 33 ++++++++++++++++++++++++++++++++- docs/sql-reference.md | 12 +++++++++++- docs/user-guide.md | 7 +++++-- pgcolumnar--1.0.sql | 13 +++++++------ 7 files changed, 72 insertions(+), 16 deletions(-) diff --git a/design/PHASE_G_DOCS_AUDIT.md b/design/PHASE_G_DOCS_AUDIT.md index 2409f1d..7475954 100644 --- a/design/PHASE_G_DOCS_AUDIT.md +++ b/design/PHASE_G_DOCS_AUDIT.md @@ -40,7 +40,7 @@ Read surfaces, all superuser, server-side paths: or FDW, directory/glob, predicate and projection pushdown, codec list. - `docs/user-guide.md`: a worked example of read_parquet and a foreign table over a directory, with an EXPLAIN showing skipping. -- `docs/ARCHITECTURE.md`: the shared scan core, the two surfaces, and where +- `docs/ARCHITECTURE.md`: the shared scan core, the three surfaces, and where pushdown sits. - `CHANGELOG.md`: entries for the read surface and the follow-ons. - `design/ROADMAP.md`: mark the Parquet read follow-ons done. diff --git a/docs/features.md b/docs/features.md index ca8caa2..2d582d3 100644 --- a/docs/features.md +++ b/docs/features.md @@ -141,6 +141,8 @@ coverage. - The foreign-table scan pushes work down: row groups whose min/max statistics exclude the query's predicate are skipped, and only the columns the query references are decoded. `EXPLAIN ANALYZE` reports the row groups read and - skipped, the columns read, and the number of files. + skipped, the columns read, and the number of files. Skipping applies to + `column op constant` clauses over integer and floating-point columns; see + [limitations.md](limitations.md) for the exact conditions. - uuid and numeric columns are read from their Parquet representations, and the reader handles millisecond, microsecond, and nanosecond time units. diff --git a/docs/installation.md b/docs/installation.md index fbea689..c31eeac 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,7 +1,7 @@ # Installation -pgColumnar builds with PGXS against an installed PostgreSQL server, versions 13 -through 19. +pgColumnar builds with PGXS against an installed PostgreSQL server, versions 15 +through 19. PostgreSQL 13 and 14 still build but are out of the tested matrix. ## Requirements @@ -10,9 +10,18 @@ through 19. - `pkg-config`. It is used to detect the optional compression libraries. - Optional: `liblz4` and `libzstd` development libraries. When present, the `lz4` and `zstd` codecs are compiled in. When absent, those codecs are compiled out - and a request for them falls back to a codec that is present. + and a request for them on a columnar table falls back to a codec that is + present. +- Optional: `zlib` development libraries. When present, the Parquet reader + decodes GZIP-compressed pages. It is not used by the native table format. +- The fallback above applies to the native table format only. When an external + Parquet file holds a page compressed with a codec that was not built in, the + read fails with a decode error rather than falling back. See + [limitations.md](limitations.md) for the codecs the reader supports. - A little-endian host is required for the Arrow and Parquet import and export - functions. The rest of the extension runs on any host PostgreSQL supports. + functions and for reading external Parquet files (`read_parquet`, + `parquet_schema`, and the `pgcolumnar_parquet` foreign-data wrapper). The rest + of the extension runs on any host PostgreSQL supports. ## Build and install diff --git a/docs/limitations.md b/docs/limitations.md index e2032d2..013c834 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -140,7 +140,7 @@ listed are rejected. | `bytea` | yes | yes | yes | yes | | `date`, `time`, `timestamp`, `timestamptz` | yes | yes | yes | yes | | `uuid` | yes | yes | yes | yes | -| `numeric` | yes | yes | yes | yes | +| `numeric` | yes | yes (`numeric(p,s)`, `p` <= 38) | yes | yes (DECIMAL only) | | `json`, `jsonb` | yes | yes | yes | no | | one-dimensional array of the above | yes | yes | yes | yes | | composite of the above | yes | yes | yes | yes | @@ -148,6 +148,13 @@ listed are rejected. `uuid` is imported from a 16-byte fixed-length binary column, and `numeric` from a DECIMAL column stored as fixed or variable big-endian bytes with precision up to 38. A DECIMAL backed by an INT32 or INT64 physical column is not yet read. + +`numeric` needs a declared precision for a Parquet round trip. The exporter +writes DECIMAL only for a column declared `numeric(p,s)` with `p` up to 38; a +`numeric` column with no precision, or one with `p` above 38, is exported as +text, and a text column cannot be imported back into `numeric`. Declare +`numeric(p,s)` with `p` up to 38 when the file has to read back into a `numeric` +column. Arrow export and import carry `numeric` in either form. `json` and `jsonb` can be exported to Parquet and read back with other tools, but pgColumnar does not currently import them; they are supported end to end through Arrow. @@ -183,3 +190,27 @@ The read-in-place surface (`read_parquet`, `parquet_schema`, and the truncated. - Each file is read fully into memory before its rows are produced; there is no streaming. +- The column definition list, or a foreign table's column list, must cover every + leaf column in the file. A shorter list is an error rather than a projection. + +Row-group skipping is narrower than the general statement that a group is skipped +when its statistics exclude the predicate. A scan that skips nothing still returns +correct rows; these are the conditions under which it can skip at all: + +- The clause must be `column op constant` with a btree comparison operator. + A parameterized qual, such as one inside a PL/pgSQL function or a generic plan + from `PREPARE`, does not drive skipping. This is deliberate: the skip set is + computed once when the scan starts and reused across rescans. +- The column must be stored as a Parquet INT32, INT64, FLOAT, or DOUBLE. Text, + bytea, uuid, numeric, and boolean columns are filtered but never skipped, + whatever their statistics. +- The constant's type must match the column's type exactly. A cross-type + comparison such as `ts >= DATE '2026-01-01'` against a `timestamp` column, or + `bigint_col > 5::int`, does not skip. +- The row group's statistics must carry both a minimum and a maximum, and the + interval must not be inverted. An unsigned Parquet column straddling the sign + boundary, or one narrowed into a smaller PostgreSQL type, decodes to an + interval that is not trusted for skipping. + +The `Row Groups Skipped` counter in `EXPLAIN ANALYZE` reports what was actually +skipped. diff --git a/docs/sql-reference.md b/docs/sql-reference.md index 315d193..1820b3a 100644 --- a/docs/sql-reference.md +++ b/docs/sql-reference.md @@ -179,6 +179,13 @@ that names the output columns and their types; the reader binds it against the file's leaf columns by position, with the same type-compatibility rules as import. +The list must cover every leaf column in the file. Declaring a subset is an +error, not a projection: the read stops with a message reporting the file's leaf +count and the count the target expands to. The same rule applies to a foreign +table's column definitions. Projection pushdown decides which of the declared +columns are decoded, which is separate from how many must be declared. Use +`parquet_schema` to generate the full list. + ```sql SELECT * FROM pgcolumnar.read_parquet('/data/events.parquet') AS t(id int, ts timestamp, amount numeric(12,2)); @@ -203,7 +210,10 @@ SELECT * FROM pgcolumnar.parquet_schema('/data/events.parquet'); Exposes a Parquet file, directory, or glob as a foreign table. The scan pushes work down: row groups whose min/max statistics exclude the query's predicate are -skipped, and only referenced columns are decoded. +skipped, and only referenced columns are decoded. Skipping requires a +`column op constant` clause over an integer or floating-point column with a +constant of the same type; [limitations.md](limitations.md) lists the conditions. +A scan that skips nothing still returns correct rows. ```sql CREATE SERVER pq FOREIGN DATA WRAPPER pgcolumnar_parquet; diff --git a/docs/user-guide.md b/docs/user-guide.md index d309567..69f0bda 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -158,8 +158,11 @@ glob pattern. `EXPLAIN (ANALYZE)` shows how much the scan skipped: ```sql EXPLAIN (ANALYZE, COSTS OFF) SELECT id FROM events WHERE ts >= '2026-01-01'; -- Foreign Scan on events --- Row Groups: 12 Row Groups Skipped: 9 --- Columns Read: 2 Columns Total: 3 Files: 4 +-- Row Groups: 12 +-- Row Groups Skipped: 9 +-- Columns Read: 2 +-- Columns Total: 3 +-- Files: 4 ``` ## Next steps diff --git a/pgcolumnar--1.0.sql b/pgcolumnar--1.0.sql index 450cac5..251a243 100644 --- a/pgcolumnar--1.0.sql +++ b/pgcolumnar--1.0.sql @@ -520,7 +520,7 @@ CREATE FUNCTION pgcolumnar.import_parquet(rel regclass, path text) AS 'MODULE_PATHNAME', 'columnar_import_parquet'; COMMENT ON FUNCTION pgcolumnar.import_parquet(regclass, text) - IS 'insert rows from a Parquet file into a table; returns rows inserted (gap 27)'; + IS 'insert rows from a Parquet file, directory, or glob into a table; returns rows inserted (gap 27)'; CREATE FUNCTION pgcolumnar.parquet_schema(path text) RETURNS TABLE(column_name text, data_type text, nullable boolean) @@ -528,7 +528,7 @@ CREATE FUNCTION pgcolumnar.parquet_schema(path text) AS 'MODULE_PATHNAME', 'columnar_parquet_schema'; COMMENT ON FUNCTION pgcolumnar.parquet_schema(text) - IS 'report the leaf columns of a Parquet file and the PostgreSQL type each maps to (Phase G scan core)'; + IS 'report the leaf columns of a Parquet file and the PostgreSQL type each maps to; for a directory or glob, of its first file (Phase G scan core)'; CREATE FUNCTION pgcolumnar.read_parquet(path text) RETURNS SETOF record @@ -536,13 +536,14 @@ CREATE FUNCTION pgcolumnar.read_parquet(path text) AS 'MODULE_PATHNAME', 'columnar_read_parquet'; COMMENT ON FUNCTION pgcolumnar.read_parquet(text) - IS 'read a Parquet file in place as a set of rows; requires a column definition list, e.g. SELECT * FROM pgcolumnar.read_parquet(path) AS t(id int, name text) (Phase G)'; + IS 'read a Parquet file, directory, or glob in place as a set of rows; requires a column definition list covering every leaf column, e.g. SELECT * FROM pgcolumnar.read_parquet(path) AS t(id int, name text) (Phase G)'; /* --------------------------------------------------------------------------- * Parquet foreign-data wrapper (Phase G) * - * A foreign table over a single Parquet file; its column definitions are bound - * against the file by position, like read_parquet's column list. Usage: + * A foreign table over a Parquet file, a directory of *.parquet files, or a glob + * pattern, read as one relation; its column definitions are bound against every + * file by position, like read_parquet's column list. Usage: * CREATE SERVER pq FOREIGN DATA WRAPPER pgcolumnar_parquet; * CREATE FOREIGN TABLE ft (id int, name text) SERVER pq * OPTIONS (path '/data/f.parquet'); @@ -563,7 +564,7 @@ CREATE FOREIGN DATA WRAPPER pgcolumnar_parquet VALIDATOR pgcolumnar.parquet_fdw_validator; COMMENT ON FOREIGN DATA WRAPPER pgcolumnar_parquet - IS 'read a single Parquet file as a foreign table; table option: path (Phase G)'; + IS 'read a Parquet file, directory, or glob as a foreign table; table option: path (Phase G)'; CREATE FUNCTION pgcolumnar.vm_selftest(rel regclass, blk int) RETURNS boolean