docs: document the external-Parquet read surface, correct stale claims

CHANGELOG.md

@@ -40,10 +40,21 @@ unreleased. For the forward-looking plan see
   one-dimensional arrays, and composite types, with nulls at every level.
 - Arrow IPC and Parquet import (`pgcolumnar.import_arrow`,
   `pgcolumnar.import_parquet`). The Parquet reader parses Thrift metadata,
-  decompresses Snappy, and decodes PLAIN and dictionary encodings from data-page
-  versions 1 and 2. Both readers reconstruct one-dimensional arrays and composite
-  types: Arrow from its List and Struct buffers, Parquet from the Dremel
-  repetition and definition levels.
+  decompresses uncompressed, Snappy, GZIP, ZSTD, and LZ4_RAW pages, and decodes
+  PLAIN and dictionary encodings from data-page versions 1 and 2. Both readers
+  reconstruct one-dimensional arrays and composite types: Arrow from its List and
+  Struct buffers, Parquet from the Dremel repetition and definition levels.
+- Reading external Parquet in place. `pgcolumnar.read_parquet(path)` returns a
+  file's rows without importing, `pgcolumnar.parquet_schema(path)` reports its
+  columns and inferred types, and the `pgcolumnar_parquet` foreign-data wrapper
+  exposes a file as a foreign table. A `path` may be a single file, a directory
+  of `*.parquet` files, or a glob pattern, read as one relation in sorted order.
+  The foreign scan skips row groups excluded by the query's predicate (min/max
+  statistics) and decodes only the referenced columns; `EXPLAIN ANALYZE` reports
+  the row groups and columns read and skipped and the number of files.
+- Parquet read type coverage extended to uuid and numeric (from fixed and
+  variable DECIMAL, precision up to 38), fixed-length binary, and millisecond,
+  microsecond, and nanosecond time units.
 - User and administrator documentation under [docs/](docs/index.md):
   installation, user guide, administration, configuration reference, SQL
   reference, and limitations.
@@ -60,6 +71,12 @@ unreleased. For the forward-looking plan see
   them, using memory proportional to the row count. They now reset a per-row
   scratch context (and, for Parquet, a per-row-group context for decoded leaf
   streams), so peak memory stays bounded on large files.
+- Hardened the Parquet reader against crafted files. File-declared page sizes,
+  DECIMAL scale, and per-row-group column-chunk counts are range-checked, so a
+  malformed footer yields a clean decode error rather than a stack overflow, an
+  out-of-bounds read, or a wrong value. Float and double row-group skipping
+  accounts for NaN and for inverted min/max intervals, and narrowing a wide
+  Parquet value into a smaller PostgreSQL type raises instead of wrapping.
 - Concurrent inserts of the same unique-index key now serialize correctly with a
   transaction-scoped advisory lock (`pgcolumnar.enable_unique_insert_lock`).
 - Lost delete marks under concurrent same-chunk-group deletes.

design/PHASE_G_DOCS_AUDIT.md

@@ -0,0 +1,62 @@
+# Phase G documentation audit
+
+Reconciles the user and engineering docs against the external-Parquet read
+surface, which landed across #98-#101 and the follow-ons #106/#107/#109/#111 and
+was never documented. Some existing claims are now false. Style per project
+convention: professional, no em-dashes, no unnecessary adjectives.
+
+## What is now true but undocumented or mis-documented
+
+Read surfaces, all superuser, server-side paths:
+- `pgcolumnar.read_parquet(path) AS t(...)` -- SRF, reads rows in place.
+- `pgcolumnar.parquet_schema(path)` -- reports leaf columns and the PostgreSQL
+  type each maps to.
+- The `pgcolumnar_parquet` foreign-data wrapper -- `CREATE SERVER` +
+  `CREATE FOREIGN TABLE ... OPTIONS (path ...)`.
+- Predicate pushdown (row-group skipping via min/max stats) and column projection
+  pushdown, both visible in `EXPLAIN ANALYZE` (Row Groups, Row Groups Skipped,
+  Columns Read/Total).
+- A `path` that is a directory reads every `*.parquet` in it; a glob pattern
+  expands; sorted, deterministic.
+- Codecs on read: uncompressed, Snappy, GZIP, ZSTD, LZ4_RAW.
+- FLBA types on read: uuid and numeric(p,s) via DECIMAL, plus fixed bytea.
+
+## Corrections (currently wrong)
+
+- `docs/limitations.md` type matrix: `uuid` and `numeric` Parquet import are
+  marked "no" -- both are "yes" now (#106). `json`/`jsonb` import stays "no".
+- `docs/limitations.md` prose "does not currently import those three types from
+  Parquet" -- now only json.
+- `docs/features.md` "decompresses Snappy" -- now uncompressed, Snappy, GZIP,
+  ZSTD, LZ4_RAW.
+- `docs/sql-reference.md` import_parquet "handles Snappy compression" -- same
+  codec list update.
+
+## Additions
+
+- `docs/sql-reference.md`: new sections for `read_parquet`, `parquet_schema`, and
+  the FDW; note directory/glob paths on all read paths; note pushdown in EXPLAIN.
+- `docs/features.md`: an "external Parquet" bullet group -- read in place via SRF
+  or FDW, directory/glob, predicate and projection pushdown, codec list.
+- `docs/user-guide.md`: a worked example of read_parquet and a foreign table over
+  a directory, with an EXPLAIN showing skipping.
+- `docs/ARCHITECTURE.md`: the shared scan core, the three surfaces, and where
+  pushdown sits.
+- `CHANGELOG.md`: entries for the read surface and the follow-ons.
+- `design/ROADMAP.md`: mark the Parquet read follow-ons done.
+
+## Still-true limitations to state
+
+- No json/jsonb import from Parquet.
+- INT32/INT64-backed DECIMAL not read (only FLBA/BYTE_ARRAY DECIMAL); the schema
+  does not advertise numeric for them.
+- TIMESTAMP_NANOS advises `bigint` (lossless); declaring `timestamp` truncates.
+- No Hive-style partition pruning, no recursive directory walk, no streaming
+  (each file is read fully into memory), single schema per directory assumed by
+  `parquet_schema`.
+- LZO, BROTLI, and the deprecated Hadoop-framed LZ4 (codec 5) are not read.
+- Reads are superuser-only and little-endian only, as export/import already are.
+
+## Order of work
+limitations (fix false claims first) -> features -> sql-reference -> user-guide
+-> ARCHITECTURE -> CHANGELOG -> ROADMAP.

design/ROADMAP.md

@@ -29,6 +29,13 @@ matrix. Gap specifications are in [gaps/](gaps/).
 | Arrow nested import (List → array, Struct → composite) | gap 27 |
 | Parquet nested import (LIST → array, group → composite; Dremel level assembly) | gap 27 |
 | **Gap 27 complete** — Arrow/Parquet interop: export + import, flat + nested, both formats | gap 27 |
+| Read external Parquet in place: `read_parquet`, `parquet_schema`, `pgcolumnar_parquet` FDW | Phase G |
+| Parquet FDW predicate pushdown (row-group skipping) and column projection pushdown | Phase G |
+| Parquet read codecs: GZIP, ZSTD, LZ4_RAW (added to uncompressed and Snappy) | Phase G |
+| Parquet read type coverage: uuid, numeric via DECIMAL, fixed binary, ms/us/ns time units | Phase G |
+| Multi-file reads: a directory of `*.parquet` or a glob read as one relation | Phase G |
+| Reader hardening against crafted files (scale/size/chunk-count guards, NaN and inverted stats) | Phase G |
+| **Phase G read surface complete** — external Parquet read, pushdown, multi-file, all matrix-gated | Phase G |
 
 ## Remaining
 
@@ -157,9 +164,11 @@ compression defaults, and the FastLanes on-disk format generation.
 The research pass returned few surviving primary sources in this area, so these
 are directions to investigate and spec, not validated recommendations:
 
-- Query external Parquet, ORC, and Arrow files with predicate and projection
-  pushdown, and read open table formats (Apache Iceberg, Delta Lake, Hudi),
-  reusing the zone-map and delete-vector machinery for file pruning.
+- External Parquet read with predicate and projection pushdown is done (Phase G,
+  see above). What remains here: ORC, open table formats (Apache Iceberg, Delta
+  Lake, Hudi), and, within Parquet, Hive-style partition pruning, recursive
+  directory walks, streaming instead of reading each file fully into memory, and
+  INT32/INT64-backed DECIMAL reads.
 - Arrow C Data Interface zero-copy export, and Arrow Flight SQL or ADBC access.
 - New PostgreSQL 17-19 integration points: read stream and asynchronous IO (partly
   used), `MERGE`, incremental materialized views (pg_ivm), logical decoding of

docs/ARCHITECTURE.md

@@ -222,17 +222,35 @@ levels (Dremel): scalars (one leaf), 1-D arrays (a 3-level LIST), and composites
 dependency; same scalar type coverage as the Arrow writer.
 
 ### columnar_parquet_reader.c
-Parquet import (`pgcolumnar.import_parquet`, gap 27). A self-contained reader: a
-Thrift compact-protocol decoder for the footer and page headers, clean-room
-Snappy decompression, the RLE/bit-packed hybrid decoder for repetition/definition
-levels and dictionary indices, PLAIN and dictionary value decoding, and both
-DATA_PAGE v1 and v2 (what pyarrow writes). The schema tree is walked to derive
-each leaf column's Dremel level bounds; nested columns are reconstructed by
-decoding each leaf's full entry sequence (defs/reps/dense values) and grouping
-repeated runs (LIST to array, group to composite), the inverse of the nested
-Parquet exporter. Scalars remain byte-for-byte the flat path. Rows are inserted
-into an existing target table via `table_tuple_insert`, mirroring the Arrow
-importer. No libparquet dependency.
+Parquet import and the external-Parquet read surface (gap 27 and Phase G). A
+self-contained reader: a Thrift compact-protocol decoder for the footer and page
+headers, clean-room Snappy plus GZIP (zlib), ZSTD, and LZ4_RAW page
+decompression, the RLE/bit-packed hybrid decoder for repetition/definition levels
+and dictionary indices, PLAIN and dictionary value decoding, and both DATA_PAGE
+v1 and v2 (what pyarrow writes). The schema tree is walked to derive each leaf
+column's Dremel level bounds; nested columns are reconstructed by decoding each
+leaf's full entry sequence (defs/reps/dense values) and grouping repeated runs
+(LIST to array, group to composite), the inverse of the nested Parquet exporter.
+Scalars remain byte-for-byte the flat path. No libparquet dependency.
+
+One shared row-producing core (`pq_read_rows`, decode one row group into slots)
+feeds three surfaces over the same file parse and type inference:
+
+- `import_parquet` inserts into a target table via `table_tuple_insert`.
+- `read_parquet` returns rows as a set-returning function, and `parquet_schema`
+  reports the leaf columns and their inferred PostgreSQL types.
+- The `pgcolumnar_parquet` foreign-data wrapper materializes the file into a
+  tuplestore drained by the scan. It pushes down predicates by skipping row
+  groups whose min/max statistics prove empty (only fixed-width ordered types,
+  with NaN and inverted-interval guards), and projects by decoding only the
+  columns the plan references (computed from the base rel's reltarget and quals).
+
+A `path` that is a directory or glob resolves to a sorted list of files, each
+read through the same core into the one sink; per-file decode buffers are freed
+between files. Decode paths are hardened against crafted files: file-declared
+sizes, DECIMAL scale, and per-row-group chunk counts are all range-checked so a
+malformed footer yields a clean error rather than an out-of-bounds read or a
+wrong value.
 
 ### columnar_visibilitymap.c
 Index-only-scan support (gap 28). A columnar visibility-map fork records which

docs/features.md

@@ -119,10 +119,30 @@ coverage.
   dependency.
 - Import from Arrow and Parquet: `pgcolumnar.import_arrow(table, path)` and
   `pgcolumnar.import_parquet(table, path)` into an existing target table. The
-  Parquet reader parses Thrift metadata, decompresses Snappy, and decodes PLAIN
-  and dictionary encodings from data-page versions 1 and 2.
+  Parquet reader parses Thrift metadata, decompresses uncompressed, Snappy, GZIP,
+  ZSTD, and LZ4_RAW pages, and decodes PLAIN and dictionary encodings from
+  data-page versions 1 and 2.
 - Both directions cover scalar types, one-dimensional arrays, and composite types
   (Arrow List and Struct, Parquet LIST and group), with nulls at every level. The
   functions require superuser and run on little-endian hosts. See the
   [SQL reference](sql-reference.md#import-and-export) and the
   [type-coverage table](limitations.md#import-and-export-type-coverage).
+
+## Reading external Parquet in place
+
+- `pgcolumnar.read_parquet(path) AS t(...)` reads a server-side Parquet file's
+  rows without importing them, and `pgcolumnar.parquet_schema(path)` reports its
+  leaf columns and the PostgreSQL type each maps to.
+- The `pgcolumnar_parquet` foreign-data wrapper exposes a Parquet file as a
+  foreign table: `CREATE FOREIGN TABLE ... SERVER ... OPTIONS (path '...')`.
+- A `path` that is a directory reads every `*.parquet` file inside it as one
+  relation, and a glob pattern expands the same way, in a deterministic sorted
+  order.
+- The foreign-table scan pushes work down: row groups whose min/max statistics
+  exclude the query's predicate are skipped, and only the columns the query
+  references are decoded. `EXPLAIN ANALYZE` reports the row groups read and
+  skipped, the columns read, and the number of files. Skipping applies to
+  `column op constant` clauses over integer and floating-point columns; see
+  [limitations.md](limitations.md) for the exact conditions.
+- uuid and numeric columns are read from their Parquet representations, and the
+  reader handles millisecond, microsecond, and nanosecond time units.

docs/installation.md

@@ -1,7 +1,7 @@
 # Installation
 
-pgColumnar builds with PGXS against an installed PostgreSQL server, versions 13
-through 19.
+pgColumnar builds with PGXS against an installed PostgreSQL server, versions 15
+through 19. PostgreSQL 13 and 14 still build but are out of the tested matrix.
 
 ## Requirements
 
@@ -10,9 +10,18 @@ through 19.
 - `pkg-config`. It is used to detect the optional compression libraries.
 - Optional: `liblz4` and `libzstd` development libraries. When present, the `lz4`
   and `zstd` codecs are compiled in. When absent, those codecs are compiled out
-  and a request for them falls back to a codec that is present.
+  and a request for them on a columnar table falls back to a codec that is
+  present.
+- Optional: `zlib` development libraries. When present, the Parquet reader
+  decodes GZIP-compressed pages. It is not used by the native table format.
+- The fallback above applies to the native table format only. When an external
+  Parquet file holds a page compressed with a codec that was not built in, the
+  read fails with a decode error rather than falling back. See
+  [limitations.md](limitations.md) for the codecs the reader supports.
 - A little-endian host is required for the Arrow and Parquet import and export
-  functions. The rest of the extension runs on any host PostgreSQL supports.
+  functions and for reading external Parquet files (`read_parquet`,
+  `parquet_schema`, and the `pgcolumnar_parquet` foreign-data wrapper). The rest
+  of the extension runs on any host PostgreSQL supports.
 
 ## Build and install
 

docs/limitations.md

@@ -139,18 +139,78 @@ listed are rejected.
 | `text`, `varchar` | yes | yes | yes | yes |
 | `bytea` | yes | yes | yes | yes |
 | `date`, `time`, `timestamp`, `timestamptz` | yes | yes | yes | yes |
-| `uuid` | yes | yes | yes | no |
-| `numeric` | yes | yes | yes | no |
+| `uuid` | yes | yes | yes | yes |
+| `numeric` | yes | yes (`numeric(p,s)`, `p` <= 38) | yes | yes (DECIMAL only) |
 | `json`, `jsonb` | yes | yes | yes | no |
 | one-dimensional array of the above | yes | yes | yes | yes |
 | composite of the above | yes | yes | yes | yes |
 
-`uuid`, `numeric`, and `json` can be exported to Parquet and read back with other
-tools, but pgColumnar does not currently import those three types from Parquet.
-They are supported end to end through Arrow.
+`uuid` is imported from a 16-byte fixed-length binary column, and `numeric` from
+a DECIMAL column stored as fixed or variable big-endian bytes with precision up
+to 38. A DECIMAL backed by an INT32 or INT64 physical column is not yet read.
+
+`numeric` needs a declared precision for a Parquet round trip. The exporter
+writes DECIMAL only for a column declared `numeric(p,s)` with `p` up to 38; a
+`numeric` column with no precision, or one with `p` above 38, is exported as
+text, and a text column cannot be imported back into `numeric`. Declare
+`numeric(p,s)` with `p` up to 38 when the file has to read back into a `numeric`
+column. Arrow export and import carry `numeric` in either form.
+`json` and `jsonb` can be exported to Parquet and read back with other tools, but
+pgColumnar does not currently import them; they are supported end to end through
+Arrow.
 
 ## Compression codecs
 
-`lz4` and `zstd` are available only when the extension was built with the
-corresponding system libraries. When a codec is not built in, a request for it
-falls back to a codec that is present. `pglz` and `none` are always available.
+For the native table format, `lz4` and `zstd` are available only when the
+extension was built with the corresponding system libraries. When a codec is not
+built in, a request for it falls back to a codec that is present. `pglz` and
+`none` are always available.
+
+When reading external Parquet files, the reader decodes uncompressed, Snappy,
+GZIP, ZSTD, and LZ4_RAW pages. GZIP requires a build with zlib, and ZSTD and
+LZ4_RAW require the same libraries as the native codecs; a page whose codec was
+not built in fails with a clean decode error. LZO, BROTLI, and the deprecated
+Hadoop-framed LZ4 (codec 5, as distinct from LZ4_RAW) are not read.
+
+## Reading external Parquet
+
+The read-in-place surface (`read_parquet`, `parquet_schema`, and the
+`pgcolumnar_parquet` foreign-data wrapper) has these limits:
+
+- Reads are superuser only and run on little-endian hosts, as import and export
+  do, since they read a server-side path.
+- A `path` that is a directory reads the `*.parquet` files directly inside it;
+  there is no recursive walk and no Hive-style partition pruning (directory names
+  of the form `col=value` are not exposed as columns).
+- `parquet_schema` describes the first file of a directory or glob, assuming the
+  set is uniform. The read paths still bind every file against the declared
+  columns, so a mismatched file raises rather than returning wrong rows.
+- A `TIMESTAMP` column with nanosecond precision is advised as `bigint`, which is
+  exact; declaring it `timestamp` reads it with the sub-microsecond digits
+  truncated.
+- Each file is read fully into memory before its rows are produced; there is no
+  streaming.
+- The column definition list, or a foreign table's column list, must cover every
+  leaf column in the file. A shorter list is an error rather than a projection.
+
+Row-group skipping is narrower than the general statement that a group is skipped
+when its statistics exclude the predicate. A scan that skips nothing still returns
+correct rows; these are the conditions under which it can skip at all:
+
+- The clause must be `column op constant` with a btree comparison operator.
+  A parameterized qual, such as one inside a PL/pgSQL function or a generic plan
+  from `PREPARE`, does not drive skipping. This is deliberate: the skip set is
+  computed once when the scan starts and reused across rescans.
+- The column must be stored as a Parquet INT32, INT64, FLOAT, or DOUBLE. Text,
+  bytea, uuid, numeric, and boolean columns are filtered but never skipped,
+  whatever their statistics.
+- The constant's type must match the column's type exactly. A cross-type
+  comparison such as `ts >= DATE '2026-01-01'` against a `timestamp` column, or
+  `bigint_col > 5::int`, does not skip.
+- The row group's statistics must carry both a minimum and a maximum, and the
+  interval must not be inverted. An unsigned Parquet column straddling the sign
+  boundary, or one narrowed into a smaller PostgreSQL type, decodes to an
+  interval that is not trusted for skipping.
+
+The `Row Groups Skipped` counter in `EXPLAIN ANALYZE` reports what was actually
+skipped.