diff --git a/CHANGELOG.md b/CHANGELOG.md index 39f5dd7..6db5ca8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,10 +40,21 @@ unreleased. For the forward-looking plan see one-dimensional arrays, and composite types, with nulls at every level. - Arrow IPC and Parquet import (`pgcolumnar.import_arrow`, `pgcolumnar.import_parquet`). The Parquet reader parses Thrift metadata, - decompresses Snappy, and decodes PLAIN and dictionary encodings from data-page - versions 1 and 2. Both readers reconstruct one-dimensional arrays and composite - types: Arrow from its List and Struct buffers, Parquet from the Dremel - repetition and definition levels. + decompresses uncompressed, Snappy, GZIP, ZSTD, and LZ4_RAW pages, and decodes + PLAIN and dictionary encodings from data-page versions 1 and 2. Both readers + reconstruct one-dimensional arrays and composite types: Arrow from its List and + Struct buffers, Parquet from the Dremel repetition and definition levels. +- Reading external Parquet in place. `pgcolumnar.read_parquet(path)` returns a + file's rows without importing, `pgcolumnar.parquet_schema(path)` reports its + columns and inferred types, and the `pgcolumnar_parquet` foreign-data wrapper + exposes a file as a foreign table. A `path` may be a single file, a directory + of `*.parquet` files, or a glob pattern, read as one relation in sorted order. + The foreign scan skips row groups excluded by the query's predicate (min/max + statistics) and decodes only the referenced columns; `EXPLAIN ANALYZE` reports + the row groups and columns read and skipped and the number of files. +- Parquet read type coverage extended to uuid and numeric (from fixed and + variable DECIMAL, precision up to 38), fixed-length binary, and millisecond, + microsecond, and nanosecond time units. - User and administrator documentation under [docs/](docs/index.md): installation, user guide, administration, configuration reference, SQL reference, and limitations. @@ -60,6 +71,12 @@ unreleased. For the forward-looking plan see them, using memory proportional to the row count. They now reset a per-row scratch context (and, for Parquet, a per-row-group context for decoded leaf streams), so peak memory stays bounded on large files. +- Hardened the Parquet reader against crafted files. File-declared page sizes, + DECIMAL scale, and per-row-group column-chunk counts are range-checked, so a + malformed footer yields a clean decode error rather than a stack overflow, an + out-of-bounds read, or a wrong value. Float and double row-group skipping + accounts for NaN and for inverted min/max intervals, and narrowing a wide + Parquet value into a smaller PostgreSQL type raises instead of wrapping. - Concurrent inserts of the same unique-index key now serialize correctly with a transaction-scoped advisory lock (`pgcolumnar.enable_unique_insert_lock`). - Lost delete marks under concurrent same-chunk-group deletes. diff --git a/design/PHASE_G_DOCS_AUDIT.md b/design/PHASE_G_DOCS_AUDIT.md new file mode 100644 index 0000000..7475954 --- /dev/null +++ b/design/PHASE_G_DOCS_AUDIT.md @@ -0,0 +1,62 @@ +# Phase G documentation audit + +Reconciles the user and engineering docs against the external-Parquet read +surface, which landed across #98-#101 and the follow-ons #106/#107/#109/#111 and +was never documented. Some existing claims are now false. Style per project +convention: professional, no em-dashes, no unnecessary adjectives. + +## What is now true but undocumented or mis-documented + +Read surfaces, all superuser, server-side paths: +- `pgcolumnar.read_parquet(path) AS t(...)` -- SRF, reads rows in place. +- `pgcolumnar.parquet_schema(path)` -- reports leaf columns and the PostgreSQL + type each maps to. +- The `pgcolumnar_parquet` foreign-data wrapper -- `CREATE SERVER` + + `CREATE FOREIGN TABLE ... OPTIONS (path ...)`. +- Predicate pushdown (row-group skipping via min/max stats) and column projection + pushdown, both visible in `EXPLAIN ANALYZE` (Row Groups, Row Groups Skipped, + Columns Read/Total). +- A `path` that is a directory reads every `*.parquet` in it; a glob pattern + expands; sorted, deterministic. +- Codecs on read: uncompressed, Snappy, GZIP, ZSTD, LZ4_RAW. +- FLBA types on read: uuid and numeric(p,s) via DECIMAL, plus fixed bytea. + +## Corrections (currently wrong) + +- `docs/limitations.md` type matrix: `uuid` and `numeric` Parquet import are + marked "no" -- both are "yes" now (#106). `json`/`jsonb` import stays "no". +- `docs/limitations.md` prose "does not currently import those three types from + Parquet" -- now only json. +- `docs/features.md` "decompresses Snappy" -- now uncompressed, Snappy, GZIP, + ZSTD, LZ4_RAW. +- `docs/sql-reference.md` import_parquet "handles Snappy compression" -- same + codec list update. + +## Additions + +- `docs/sql-reference.md`: new sections for `read_parquet`, `parquet_schema`, and + the FDW; note directory/glob paths on all read paths; note pushdown in EXPLAIN. +- `docs/features.md`: an "external Parquet" bullet group -- read in place via SRF + or FDW, directory/glob, predicate and projection pushdown, codec list. +- `docs/user-guide.md`: a worked example of read_parquet and a foreign table over + a directory, with an EXPLAIN showing skipping. +- `docs/ARCHITECTURE.md`: the shared scan core, the three surfaces, and where + pushdown sits. +- `CHANGELOG.md`: entries for the read surface and the follow-ons. +- `design/ROADMAP.md`: mark the Parquet read follow-ons done. + +## Still-true limitations to state + +- No json/jsonb import from Parquet. +- INT32/INT64-backed DECIMAL not read (only FLBA/BYTE_ARRAY DECIMAL); the schema + does not advertise numeric for them. +- TIMESTAMP_NANOS advises `bigint` (lossless); declaring `timestamp` truncates. +- No Hive-style partition pruning, no recursive directory walk, no streaming + (each file is read fully into memory), single schema per directory assumed by + `parquet_schema`. +- LZO, BROTLI, and the deprecated Hadoop-framed LZ4 (codec 5) are not read. +- Reads are superuser-only and little-endian only, as export/import already are. + +## Order of work +limitations (fix false claims first) -> features -> sql-reference -> user-guide +-> ARCHITECTURE -> CHANGELOG -> ROADMAP. diff --git a/design/ROADMAP.md b/design/ROADMAP.md index caf219b..d9190ea 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -29,6 +29,13 @@ matrix. Gap specifications are in [gaps/](gaps/). | Arrow nested import (List → array, Struct → composite) | gap 27 | | Parquet nested import (LIST → array, group → composite; Dremel level assembly) | gap 27 | | **Gap 27 complete** — Arrow/Parquet interop: export + import, flat + nested, both formats | gap 27 | +| Read external Parquet in place: `read_parquet`, `parquet_schema`, `pgcolumnar_parquet` FDW | Phase G | +| Parquet FDW predicate pushdown (row-group skipping) and column projection pushdown | Phase G | +| Parquet read codecs: GZIP, ZSTD, LZ4_RAW (added to uncompressed and Snappy) | Phase G | +| Parquet read type coverage: uuid, numeric via DECIMAL, fixed binary, ms/us/ns time units | Phase G | +| Multi-file reads: a directory of `*.parquet` or a glob read as one relation | Phase G | +| Reader hardening against crafted files (scale/size/chunk-count guards, NaN and inverted stats) | Phase G | +| **Phase G read surface complete** — external Parquet read, pushdown, multi-file, all matrix-gated | Phase G | ## Remaining @@ -157,9 +164,11 @@ compression defaults, and the FastLanes on-disk format generation. The research pass returned few surviving primary sources in this area, so these are directions to investigate and spec, not validated recommendations: -- Query external Parquet, ORC, and Arrow files with predicate and projection - pushdown, and read open table formats (Apache Iceberg, Delta Lake, Hudi), - reusing the zone-map and delete-vector machinery for file pruning. +- External Parquet read with predicate and projection pushdown is done (Phase G, + see above). What remains here: ORC, open table formats (Apache Iceberg, Delta + Lake, Hudi), and, within Parquet, Hive-style partition pruning, recursive + directory walks, streaming instead of reading each file fully into memory, and + INT32/INT64-backed DECIMAL reads. - Arrow C Data Interface zero-copy export, and Arrow Flight SQL or ADBC access. - New PostgreSQL 17-19 integration points: read stream and asynchronous IO (partly used), `MERGE`, incremental materialized views (pg_ivm), logical decoding of diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ea8a63c..b7c952d 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -222,17 +222,35 @@ levels (Dremel): scalars (one leaf), 1-D arrays (a 3-level LIST), and composites dependency; same scalar type coverage as the Arrow writer. ### columnar_parquet_reader.c -Parquet import (`pgcolumnar.import_parquet`, gap 27). A self-contained reader: a -Thrift compact-protocol decoder for the footer and page headers, clean-room -Snappy decompression, the RLE/bit-packed hybrid decoder for repetition/definition -levels and dictionary indices, PLAIN and dictionary value decoding, and both -DATA_PAGE v1 and v2 (what pyarrow writes). The schema tree is walked to derive -each leaf column's Dremel level bounds; nested columns are reconstructed by -decoding each leaf's full entry sequence (defs/reps/dense values) and grouping -repeated runs (LIST to array, group to composite), the inverse of the nested -Parquet exporter. Scalars remain byte-for-byte the flat path. Rows are inserted -into an existing target table via `table_tuple_insert`, mirroring the Arrow -importer. No libparquet dependency. +Parquet import and the external-Parquet read surface (gap 27 and Phase G). A +self-contained reader: a Thrift compact-protocol decoder for the footer and page +headers, clean-room Snappy plus GZIP (zlib), ZSTD, and LZ4_RAW page +decompression, the RLE/bit-packed hybrid decoder for repetition/definition levels +and dictionary indices, PLAIN and dictionary value decoding, and both DATA_PAGE +v1 and v2 (what pyarrow writes). The schema tree is walked to derive each leaf +column's Dremel level bounds; nested columns are reconstructed by decoding each +leaf's full entry sequence (defs/reps/dense values) and grouping repeated runs +(LIST to array, group to composite), the inverse of the nested Parquet exporter. +Scalars remain byte-for-byte the flat path. No libparquet dependency. + +One shared row-producing core (`pq_read_rows`, decode one row group into slots) +feeds three surfaces over the same file parse and type inference: + +- `import_parquet` inserts into a target table via `table_tuple_insert`. +- `read_parquet` returns rows as a set-returning function, and `parquet_schema` + reports the leaf columns and their inferred PostgreSQL types. +- The `pgcolumnar_parquet` foreign-data wrapper materializes the file into a + tuplestore drained by the scan. It pushes down predicates by skipping row + groups whose min/max statistics prove empty (only fixed-width ordered types, + with NaN and inverted-interval guards), and projects by decoding only the + columns the plan references (computed from the base rel's reltarget and quals). + +A `path` that is a directory or glob resolves to a sorted list of files, each +read through the same core into the one sink; per-file decode buffers are freed +between files. Decode paths are hardened against crafted files: file-declared +sizes, DECIMAL scale, and per-row-group chunk counts are all range-checked so a +malformed footer yields a clean error rather than an out-of-bounds read or a +wrong value. ### columnar_visibilitymap.c Index-only-scan support (gap 28). A columnar visibility-map fork records which diff --git a/docs/features.md b/docs/features.md index 0cff194..2d582d3 100644 --- a/docs/features.md +++ b/docs/features.md @@ -119,10 +119,30 @@ coverage. dependency. - Import from Arrow and Parquet: `pgcolumnar.import_arrow(table, path)` and `pgcolumnar.import_parquet(table, path)` into an existing target table. The - Parquet reader parses Thrift metadata, decompresses Snappy, and decodes PLAIN - and dictionary encodings from data-page versions 1 and 2. + Parquet reader parses Thrift metadata, decompresses uncompressed, Snappy, GZIP, + ZSTD, and LZ4_RAW pages, and decodes PLAIN and dictionary encodings from + data-page versions 1 and 2. - Both directions cover scalar types, one-dimensional arrays, and composite types (Arrow List and Struct, Parquet LIST and group), with nulls at every level. The functions require superuser and run on little-endian hosts. See the [SQL reference](sql-reference.md#import-and-export) and the [type-coverage table](limitations.md#import-and-export-type-coverage). + +## Reading external Parquet in place + +- `pgcolumnar.read_parquet(path) AS t(...)` reads a server-side Parquet file's + rows without importing them, and `pgcolumnar.parquet_schema(path)` reports its + leaf columns and the PostgreSQL type each maps to. +- The `pgcolumnar_parquet` foreign-data wrapper exposes a Parquet file as a + foreign table: `CREATE FOREIGN TABLE ... SERVER ... OPTIONS (path '...')`. +- A `path` that is a directory reads every `*.parquet` file inside it as one + relation, and a glob pattern expands the same way, in a deterministic sorted + order. +- The foreign-table scan pushes work down: row groups whose min/max statistics + exclude the query's predicate are skipped, and only the columns the query + references are decoded. `EXPLAIN ANALYZE` reports the row groups read and + skipped, the columns read, and the number of files. Skipping applies to + `column op constant` clauses over integer and floating-point columns; see + [limitations.md](limitations.md) for the exact conditions. +- uuid and numeric columns are read from their Parquet representations, and the + reader handles millisecond, microsecond, and nanosecond time units. diff --git a/docs/installation.md b/docs/installation.md index fbea689..c31eeac 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,7 +1,7 @@ # Installation -pgColumnar builds with PGXS against an installed PostgreSQL server, versions 13 -through 19. +pgColumnar builds with PGXS against an installed PostgreSQL server, versions 15 +through 19. PostgreSQL 13 and 14 still build but are out of the tested matrix. ## Requirements @@ -10,9 +10,18 @@ through 19. - `pkg-config`. It is used to detect the optional compression libraries. - Optional: `liblz4` and `libzstd` development libraries. When present, the `lz4` and `zstd` codecs are compiled in. When absent, those codecs are compiled out - and a request for them falls back to a codec that is present. + and a request for them on a columnar table falls back to a codec that is + present. +- Optional: `zlib` development libraries. When present, the Parquet reader + decodes GZIP-compressed pages. It is not used by the native table format. +- The fallback above applies to the native table format only. When an external + Parquet file holds a page compressed with a codec that was not built in, the + read fails with a decode error rather than falling back. See + [limitations.md](limitations.md) for the codecs the reader supports. - A little-endian host is required for the Arrow and Parquet import and export - functions. The rest of the extension runs on any host PostgreSQL supports. + functions and for reading external Parquet files (`read_parquet`, + `parquet_schema`, and the `pgcolumnar_parquet` foreign-data wrapper). The rest + of the extension runs on any host PostgreSQL supports. ## Build and install diff --git a/docs/limitations.md b/docs/limitations.md index a9ab38f..013c834 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -139,18 +139,78 @@ listed are rejected. | `text`, `varchar` | yes | yes | yes | yes | | `bytea` | yes | yes | yes | yes | | `date`, `time`, `timestamp`, `timestamptz` | yes | yes | yes | yes | -| `uuid` | yes | yes | yes | no | -| `numeric` | yes | yes | yes | no | +| `uuid` | yes | yes | yes | yes | +| `numeric` | yes | yes (`numeric(p,s)`, `p` <= 38) | yes | yes (DECIMAL only) | | `json`, `jsonb` | yes | yes | yes | no | | one-dimensional array of the above | yes | yes | yes | yes | | composite of the above | yes | yes | yes | yes | -`uuid`, `numeric`, and `json` can be exported to Parquet and read back with other -tools, but pgColumnar does not currently import those three types from Parquet. -They are supported end to end through Arrow. +`uuid` is imported from a 16-byte fixed-length binary column, and `numeric` from +a DECIMAL column stored as fixed or variable big-endian bytes with precision up +to 38. A DECIMAL backed by an INT32 or INT64 physical column is not yet read. + +`numeric` needs a declared precision for a Parquet round trip. The exporter +writes DECIMAL only for a column declared `numeric(p,s)` with `p` up to 38; a +`numeric` column with no precision, or one with `p` above 38, is exported as +text, and a text column cannot be imported back into `numeric`. Declare +`numeric(p,s)` with `p` up to 38 when the file has to read back into a `numeric` +column. Arrow export and import carry `numeric` in either form. +`json` and `jsonb` can be exported to Parquet and read back with other tools, but +pgColumnar does not currently import them; they are supported end to end through +Arrow. ## Compression codecs -`lz4` and `zstd` are available only when the extension was built with the -corresponding system libraries. When a codec is not built in, a request for it -falls back to a codec that is present. `pglz` and `none` are always available. +For the native table format, `lz4` and `zstd` are available only when the +extension was built with the corresponding system libraries. When a codec is not +built in, a request for it falls back to a codec that is present. `pglz` and +`none` are always available. + +When reading external Parquet files, the reader decodes uncompressed, Snappy, +GZIP, ZSTD, and LZ4_RAW pages. GZIP requires a build with zlib, and ZSTD and +LZ4_RAW require the same libraries as the native codecs; a page whose codec was +not built in fails with a clean decode error. LZO, BROTLI, and the deprecated +Hadoop-framed LZ4 (codec 5, as distinct from LZ4_RAW) are not read. + +## Reading external Parquet + +The read-in-place surface (`read_parquet`, `parquet_schema`, and the +`pgcolumnar_parquet` foreign-data wrapper) has these limits: + +- Reads are superuser only and run on little-endian hosts, as import and export + do, since they read a server-side path. +- A `path` that is a directory reads the `*.parquet` files directly inside it; + there is no recursive walk and no Hive-style partition pruning (directory names + of the form `col=value` are not exposed as columns). +- `parquet_schema` describes the first file of a directory or glob, assuming the + set is uniform. The read paths still bind every file against the declared + columns, so a mismatched file raises rather than returning wrong rows. +- A `TIMESTAMP` column with nanosecond precision is advised as `bigint`, which is + exact; declaring it `timestamp` reads it with the sub-microsecond digits + truncated. +- Each file is read fully into memory before its rows are produced; there is no + streaming. +- The column definition list, or a foreign table's column list, must cover every + leaf column in the file. A shorter list is an error rather than a projection. + +Row-group skipping is narrower than the general statement that a group is skipped +when its statistics exclude the predicate. A scan that skips nothing still returns +correct rows; these are the conditions under which it can skip at all: + +- The clause must be `column op constant` with a btree comparison operator. + A parameterized qual, such as one inside a PL/pgSQL function or a generic plan + from `PREPARE`, does not drive skipping. This is deliberate: the skip set is + computed once when the scan starts and reused across rescans. +- The column must be stored as a Parquet INT32, INT64, FLOAT, or DOUBLE. Text, + bytea, uuid, numeric, and boolean columns are filtered but never skipped, + whatever their statistics. +- The constant's type must match the column's type exactly. A cross-type + comparison such as `ts >= DATE '2026-01-01'` against a `timestamp` column, or + `bigint_col > 5::int`, does not skip. +- The row group's statistics must carry both a minimum and a maximum, and the + interval must not be inverted. An unsigned Parquet column straddling the sign + boundary, or one narrowed into a smaller PostgreSQL type, decodes to an + interval that is not trusted for skipping. + +The `Row Groups Skipped` counter in `EXPLAIN ANALYZE` reports what was actually +skipped. diff --git a/docs/sql-reference.md b/docs/sql-reference.md index d3af37d..1820b3a 100644 --- a/docs/sql-reference.md +++ b/docs/sql-reference.md @@ -150,14 +150,81 @@ rows inserted. ### pgcolumnar.import_parquet(rel regclass, path text) returns bigint Inserts the rows of a Parquet file at `path` into the existing table `rel`. The -reader handles Snappy compression, PLAIN and dictionary encodings, and data-page -versions 1 and 2. Returns the number of rows inserted. +reader handles uncompressed, Snappy, GZIP, ZSTD, and LZ4_RAW pages, PLAIN and +dictionary encodings, and data-page versions 1 and 2. `path` may name a single +file, a directory (every `*.parquet` file inside it is imported), or a glob +pattern. Returns the number of rows inserted. ```sql -- round-trip a table through Parquet SELECT pgcolumnar.export_parquet('events', '/tmp/events.parquet'); -- returns row count CREATE TABLE events_copy (LIKE events) USING pgcolumnar; SELECT pgcolumnar.import_parquet('events_copy', '/tmp/events.parquet'); + +-- import an entire directory of Parquet files +SELECT pgcolumnar.import_parquet('events_copy', '/data/events/'); +``` + +## Reading external Parquet + +These read a server-side Parquet file in place, without importing it. They +require superuser and run on little-endian hosts. In every case `path` may be a +single file, a directory (all `*.parquet` files inside it, read as one relation +in sorted order), or a glob pattern. + +### pgcolumnar.read_parquet(path text) returns setof record + +Returns the rows of a Parquet file. The caller supplies a column definition list +that names the output columns and their types; the reader binds it against the +file's leaf columns by position, with the same type-compatibility rules as +import. + +The list must cover every leaf column in the file. Declaring a subset is an +error, not a projection: the read stops with a message reporting the file's leaf +count and the count the target expands to. The same rule applies to a foreign +table's column definitions. Projection pushdown decides which of the declared +columns are decoded, which is separate from how many must be declared. Use +`parquet_schema` to generate the full list. + +```sql +SELECT * FROM pgcolumnar.read_parquet('/data/events.parquet') + AS t(id int, ts timestamp, amount numeric(12,2)); + +-- read a whole directory +SELECT count(*) FROM pgcolumnar.read_parquet('/data/events/') + AS t(id int, ts timestamp, amount numeric(12,2)); +``` + +### pgcolumnar.parquet_schema(path text) returns table(column_name text, data_type text, nullable bool) + +Reports the leaf columns of a Parquet file and the PostgreSQL type each maps to, +without reading the data. Useful for writing the column definition list for +`read_parquet` or a foreign table. For a directory or glob it describes the first +file. + +```sql +SELECT * FROM pgcolumnar.parquet_schema('/data/events.parquet'); +``` + +### The pgcolumnar_parquet foreign-data wrapper + +Exposes a Parquet file, directory, or glob as a foreign table. The scan pushes +work down: row groups whose min/max statistics exclude the query's predicate are +skipped, and only referenced columns are decoded. Skipping requires a +`column op constant` clause over an integer or floating-point column with a +constant of the same type; [limitations.md](limitations.md) lists the conditions. +A scan that skips nothing still returns correct rows. + +```sql +CREATE SERVER pq FOREIGN DATA WRAPPER pgcolumnar_parquet; +CREATE FOREIGN TABLE events (id int, ts timestamp, amount numeric(12,2)) + SERVER pq OPTIONS (path '/data/events/'); + +SELECT sum(amount) FROM events WHERE ts >= '2026-01-01'; + +-- EXPLAIN ANALYZE reports Row Groups, Row Groups Skipped, Columns Read, +-- Columns Total, and Files. +EXPLAIN (ANALYZE, COSTS OFF) SELECT id FROM events WHERE ts >= '2026-01-01'; ``` ## Visibility map inspection diff --git a/docs/testing.md b/docs/testing.md index db94832..97b0aa4 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -31,6 +31,16 @@ test/arrow_nested.sh /path/to/pg_config # nested Arrow export test/parquet_nested.sh /path/to/pg_config # nested Parquet export test/arrow_nested_import.sh /path/to/pg_config # nested Arrow import test/parquet_nested_import.sh /path/to/pg_config # nested Parquet import +test/native_parquet_schema.sh /path/to/pg_config # parquet_schema type inference +test/native_read_parquet.sh /path/to/pg_config # read_parquet in place +test/native_parquet_fdw.sh /path/to/pg_config # pgcolumnar_parquet foreign tables +test/native_parquet_pushdown.sh /path/to/pg_config # FDW row-group predicate skipping +test/native_parquet_projection.sh /path/to/pg_config # FDW column projection pushdown +test/native_parquet_units.sh /path/to/pg_config # TIME/TIMESTAMP unit handling +test/native_parquet_flba.sh /path/to/pg_config # uuid, decimal, fixed binary reads +test/native_parquet_codecs.sh /path/to/pg_config # GZIP, ZSTD, LZ4_RAW page reads +test/native_parquet_hardening.sh /path/to/pg_config # crafted-file decode guards +test/native_parquet_multifile.sh /path/to/pg_config # directory and glob reads test/native_writer.sh /path/to/pg_config # native format catalog output test/native_roundtrip.sh /path/to/pg_config # native write then read round-trip test/native_encoding.sh /path/to/pg_config # native per-vector encoding cascade diff --git a/docs/user-guide.md b/docs/user-guide.md index afcbb30..69f0bda 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -126,6 +126,45 @@ CREATE TABLE people ( INSERT INTO people VALUES (1, ARRAY['a','b'], ROW('Portland','97201')::addr); ``` +## Read Parquet files in place + +You can query a server-side Parquet file without importing it, either as a +set-returning function or as a foreign table. Both require superuser. + +```sql +-- inspect the file's columns and their inferred types +SELECT * FROM pgcolumnar.parquet_schema('/data/events.parquet'); + +-- read the rows directly +SELECT count(*), max(ts) +FROM pgcolumnar.read_parquet('/data/events.parquet') + AS t(id int, ts timestamp, amount numeric(12,2)); +``` + +For repeated queries, a foreign table is more convenient, and its scans skip row +groups and columns the query does not need: + +```sql +CREATE SERVER pq FOREIGN DATA WRAPPER pgcolumnar_parquet; +CREATE FOREIGN TABLE events (id int, ts timestamp, amount numeric(12,2)) + SERVER pq OPTIONS (path '/data/events/'); -- a directory of *.parquet files + +SELECT sum(amount) FROM events WHERE ts >= '2026-01-01'; +``` + +A `path` may name one file, a directory (all `*.parquet` files inside it), or a +glob pattern. `EXPLAIN (ANALYZE)` shows how much the scan skipped: + +```sql +EXPLAIN (ANALYZE, COSTS OFF) SELECT id FROM events WHERE ts >= '2026-01-01'; +-- Foreign Scan on events +-- Row Groups: 12 +-- Row Groups Skipped: 9 +-- Columns Read: 2 +-- Columns Total: 3 +-- Files: 4 +``` + ## Next steps - Operate tables in production: [Administration](administration.md). diff --git a/pgcolumnar--1.0.sql b/pgcolumnar--1.0.sql index 450cac5..251a243 100644 --- a/pgcolumnar--1.0.sql +++ b/pgcolumnar--1.0.sql @@ -520,7 +520,7 @@ CREATE FUNCTION pgcolumnar.import_parquet(rel regclass, path text) AS 'MODULE_PATHNAME', 'columnar_import_parquet'; COMMENT ON FUNCTION pgcolumnar.import_parquet(regclass, text) - IS 'insert rows from a Parquet file into a table; returns rows inserted (gap 27)'; + IS 'insert rows from a Parquet file, directory, or glob into a table; returns rows inserted (gap 27)'; CREATE FUNCTION pgcolumnar.parquet_schema(path text) RETURNS TABLE(column_name text, data_type text, nullable boolean) @@ -528,7 +528,7 @@ CREATE FUNCTION pgcolumnar.parquet_schema(path text) AS 'MODULE_PATHNAME', 'columnar_parquet_schema'; COMMENT ON FUNCTION pgcolumnar.parquet_schema(text) - IS 'report the leaf columns of a Parquet file and the PostgreSQL type each maps to (Phase G scan core)'; + IS 'report the leaf columns of a Parquet file and the PostgreSQL type each maps to; for a directory or glob, of its first file (Phase G scan core)'; CREATE FUNCTION pgcolumnar.read_parquet(path text) RETURNS SETOF record @@ -536,13 +536,14 @@ CREATE FUNCTION pgcolumnar.read_parquet(path text) AS 'MODULE_PATHNAME', 'columnar_read_parquet'; COMMENT ON FUNCTION pgcolumnar.read_parquet(text) - IS 'read a Parquet file in place as a set of rows; requires a column definition list, e.g. SELECT * FROM pgcolumnar.read_parquet(path) AS t(id int, name text) (Phase G)'; + IS 'read a Parquet file, directory, or glob in place as a set of rows; requires a column definition list covering every leaf column, e.g. SELECT * FROM pgcolumnar.read_parquet(path) AS t(id int, name text) (Phase G)'; /* --------------------------------------------------------------------------- * Parquet foreign-data wrapper (Phase G) * - * A foreign table over a single Parquet file; its column definitions are bound - * against the file by position, like read_parquet's column list. Usage: + * A foreign table over a Parquet file, a directory of *.parquet files, or a glob + * pattern, read as one relation; its column definitions are bound against every + * file by position, like read_parquet's column list. Usage: * CREATE SERVER pq FOREIGN DATA WRAPPER pgcolumnar_parquet; * CREATE FOREIGN TABLE ft (id int, name text) SERVER pq * OPTIONS (path '/data/f.parquet'); @@ -563,7 +564,7 @@ CREATE FOREIGN DATA WRAPPER pgcolumnar_parquet VALIDATOR pgcolumnar.parquet_fdw_validator; COMMENT ON FOREIGN DATA WRAPPER pgcolumnar_parquet - IS 'read a single Parquet file as a foreign table; table option: path (Phase G)'; + IS 'read a Parquet file, directory, or glob as a foreign table; table option: path (Phase G)'; CREATE FUNCTION pgcolumnar.vm_selftest(rel regclass, blk int) RETURNS boolean