From bb8329a9fb845a0125c78379b185501985827db5 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 27 Jul 2026 09:15:45 -0600 Subject: [PATCH] docs: refresh against the tree, and a full benchmark run that found a regression All three harnesses re-run at 7a9c9f7 on an idle box, and the documentation brought back in line with what the code now does. The run found something. Point lookup measured 1251.88 ms against 23.75 ms in the previous run, on the same machine and harness. It is not the lazy-decoding slot: an A/B across that merge on an unanalyzed table gives 16.42 ms before and 13.73 ms after. The difference between that probe and the harness is that the harness runs ANALYZE first, so the planner is choosing differently once statistics exist. Filed as #171, which also carries a second observation from the same investigation: ANALYZE ran for 23 minutes on a 250,000-row 8-column table and 40 minutes on a 6,000,000-row one in isolation, without reproducing inside the benchmark, which is why the issue says what is confirmed and what is not rather than naming a cause. The document presents that row as a bug report, not as a property of the design. Everything else moved the right way. DELETE of 1000 rows by id range is 14.7 ms against 22.8, a row-ordered UPDATE 14.28 against 20.78, both from #160 and #164. Two larger wins are recorded but not in the harness: index-driven access to a wide table went from 1,001,374 ms to 614 ms for 2,000 fetches on 41 columns when the lazy slot landed, and ANALYZE now collects statistics at all. Documentation. limitations.md said a point lookup must decode the whole row group and that bulk UPDATE is proportional to rows times group size; both were made false by #143 and #169 and now describe what happens. It gains what an import does about indexes and constraints, the deferrable-unique difference tracked as #168, and the ANALYZE cost with a pointer to #171. features.md gains column statistics and needed-columns fetch. ROADMAP's remaining list was "complete as of 2026-07-23" and is now the actual open set, ordered. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_012uKWWwBDt5TWWS5DR2tzDb --- bench/sample_output_all_2026_07_27.txt | 291 +++++++++++++++++++++++++ design/ROADMAP.md | 36 ++- docs/benchmarks.md | 158 +++++++------- docs/features.md | 11 +- docs/limitations.md | 37 +++- 5 files changed, 436 insertions(+), 97 deletions(-) create mode 100644 bench/sample_output_all_2026_07_27.txt diff --git a/bench/sample_output_all_2026_07_27.txt b/bench/sample_output_all_2026_07_27.txt new file mode 100644 index 0000000..0cac0d2 --- /dev/null +++ b/bench/sample_output_all_2026_07_27.txt @@ -0,0 +1,291 @@ +commit: 7a9c9f7 +=== MAIN SUITE === +started 2026-07-27T12:49:19+00:00 +== pgColumnar benchmark == +PG_CONFIG=/usr/local/pg17_nc/bin/pg_config (PostgreSQL 17.10) +scale=6000000 rows reps=5 workdir=/tmp/pgcolumnar-bench.vqC1qE +-- building (non-assert expected) +-- initdb and start +-- loading 6000000 rows (staging, heap, columnar zstd, columnar none) +-- running queries + +=== TABLE SIZES === + table | total_size | bytes +------------------------+------------+----------- + heap | 707 MB | 741793792 + columnar (zstd) | 135 MB | 141033472 + columnar (none) | 40 MB | 42287104 +(3 rows) + + +(table-only size, excluding indexes) + table | table_size +------------------------+------------ + heap | 579 MB + columnar (none) | 40 MB + columnar (zstd) | 6096 kB +(3 rows) + + +=== QUERY LATENCY: heap vs columnar zstd (median ms of 5) === + id | query | heap_ms | columnar_ms | heap_over_col +----+----------------------------------------+---------+-------------+--------------- + 1 | count(*) full table | 254.28 | 0.02 | 12714.00 + 2 | sum/avg over one int column | 379.89 | 0.56 | 678.38 + 3 | filtered agg, min/max-skippable range | 268.96 | 90.70 | 2.97 + 4 | point lookup by indexed id | 0.01 | 1251.88 | 0.00 + 5 | projection: 3 of 8 cols, 1% filter | 232.83 | 85.83 | 2.71 +(5 rows) + + +=== VECTORIZATION on vs off (columnar zstd, median ms) === + query | on_ms | off_ms | speedup_vec +----------------------+-------+---------+------------- + sum/avg over int | 0.51 | 1392.60 | 2730.59 + filtered agg (range) | 92.12 | 90.32 | 0.98 +(2 rows) + + +=== COMPRESSION none vs zstd (size and scan latency) === + metric | none | zstd | none_over_zstd +------------+-------+---------+---------------- + total size | 40 MB | 6096 kB | 6.77 +(1 row) + + metric | none | zstd | none_over_zstd +-----------------+------+------+---------------- + sum/avg scan ms | 0.52 | 0.52 | + count(*) ms | 0.02 | 0.02 | +(2 rows) + +-- sorted projection +Pager usage is off. +SET +CREATE TABLE +INSERT 0 6000000 +ANALYZE + +=== SORTED PROJECTION: narrow range scan on a scattered key (median ms) === + state | ms +----------------------+-------- + before vacuum_sorted | 364.13 +(1 row) + + vacuum_sorted +--------------- + +(1 row) + +ANALYZE + state | ms +---------------------+------- + after vacuum_sorted | 47.72 +(1 row) + +-- index-only scan +Pager usage is off. +SET +SET +SET +CREATE TABLE +INSERT 0 6000000 +CREATE INDEX +VACUUM +ANALYZE + +=== INDEX-ONLY SCAN on vs off (covering range count, median ms) === + query | on_ms | off_ms | speedup_ios +--------------------------------+-------+--------+------------- + covering count, id range (~2%) | 7.53 | 698.95 | 92.82 +(1 row) + +SET +-- projection scan +Pager usage is off. +SET +CREATE TABLE + add_projection +---------------- + +(1 row) + +INSERT 0 6000000 +ANALYZE + +=== PROJECTION SCAN on vs off (covering row scan on a scattered sort key, median ms) === + query | on_ms | off_ms | speedup_proj +--------------------------------------+--------+--------+-------------- + sortk,val where sortk in ~0.1% range | 191.41 | 635.35 | 3.32 +(1 row) + +SET +-- export +CREATE TABLE +INSERT 0 6000000 +CREATE FUNCTION + +=== EXPORT: Arrow IPC and Parquet (6000000 rows, 5 columns) === + format | ms | file_size | m_rows_per_s +----------+--------+-----------+-------------- + arrow | 1008.6 | 186 MB | 5.9 + parquet | 1100.2 | 186 MB | 5.5 +(2 rows) + +-- import +Pager usage is off. +CREATE TABLE +CREATE TABLE +CREATE FUNCTION + +=== IMPORT: Arrow IPC and Parquet (6000000 rows, 5 columns) === + format | ms | m_rows_per_s +----------+---------+-------------- + arrow | 17721.5 | 0.3 + parquet | 17762.5 | 0.3 +(2 rows) + +(row-count check: expect 6000000 for both) + arrow_rows | parquet_rows +------------+-------------- + 6000000 | 6000000 +(1 row) + +-- nested export/import +Pager usage is off. +NOTICE: type "bnc" does not exist, skipping +DROP TYPE +CREATE TYPE +CREATE TABLE +INSERT 0 1000000 +CREATE TABLE +CREATE TABLE +CREATE FUNCTION + +=== NESTED export + import (1000000 rows: int[3] array + composite) === + format | export_ms | import_ms | file_size +----------+-----------+-----------+----------- + arrow | 622.8 | 4904.1 | 38 MB + parquet | 533.7 | 4860.1 | 35 MB +(2 rows) + +(round-trip check: rows in source but not reconstructed; expect 0/0) + arrow_diff | parquet_diff +------------+-------------- + 0 | 0 +(1 row) + +-- mutation (1000000 rows) + +=== MUTATION: rows reached by index (1000000 rows) === +(median of 5 for UPDATE; DELETE is a single run, since a repeat +would find nothing left to delete) + operation | heap_ms | columnar_ms | columnar_over_heap +------------------------------------+---------+-------------+-------------------- + single row by id | 0.02 | 0.22 | 11.00 + 1000 rows, ids in row order | 3.81 | 14.28 | 3.75 + 1000 rows, ids scattered | 43.15 | 147.89 | 3.43 +(3 rows) + + + operation | heap_ms | columnar_ms | columnar_over_heap +------------------------------------+---------+-------------+-------------------- + DELETE 1000 rows by id range | 0.5 | 14.7 | 29.40 +(1 row) + +(row counts must agree after the mutations) + heap_rows | columnar_rows +-----------+--------------- + 999000 | 999000 +(1 row) + + +=== DuckDB comparison (same data, in-process columnar engine) === +Run Time (s): real 0.330 user 0.983363 sys 0.180577 +┌──────────────┐ +│ count_star() │ +│ int64 │ +├──────────────┤ +│ 6000000 │ +└──────────────┘ +Run Time (s): real 0.001 user 0.000100 sys 0.002808 +┌─────────────┬──────────┐ +│ sum(val) │ avg(val) │ +│ int128 │ double │ +├─────────────┼──────────┤ +│ 29997000000 │ 4999.5 │ +└─────────────┴──────────┘ +Run Time (s): real 0.004 user 0.013057 sys 0.000031 +┌───────────┐ +│ sum(val) │ +│ int128 │ +├───────────┤ +│ 599940000 │ +└───────────┘ +Run Time (s): real 0.001 user 0.001897 sys 0.000000 + +=== Cross-engine read of pgColumnar's Parquet output (same file) === +┌─────────┬─────────────┐ +│ rows │ sum_val │ +│ int64 │ int128 │ +├─────────┼─────────────┤ +│ 6000000 │ 29997000000 │ +└─────────┴─────────────┘ +Run Time (s): real 0.012 user 0.031017 sys 0.006628 +pyarrow read_table: rows=6000000 sum_val=29997000000 149.4 ms + +== benchmark complete == +MAIN_EXIT=0 +=== FSST === +== pgColumnar FSST ingestion micro-benchmark == +PG_CONFIG=/usr/local/pg17_nc/bin/pg_config (PostgreSQL 17.10) +scale=3000000 rows workdir=/tmp/pgcolumnar-fsstbench.WqMqsy +-- building (non-assert expected) +-- initdb and start +-- staging 3000000 rows of shared-substring text +-- timing heap ingestion (baseline) +-- timing columnar (compression=none) ingestion + alter_columnar_table_set +-------------------------- + +(1 row) + +-- verify round-trip (heap vs columnar fingerprint) +-- sizes and FSST usage + +===================== FSST INGESTION GATE ====================== + rows 3000000 + heap INSERT 2003.8 ms + columnar INSERT 12855.6 ms + round-trip MATCH + heap size 439279616 bytes + columnar size 105807872 bytes + url chunks using FSST 20 / 20 (first-vector = FSST) +================================================================ +Run once as shipped and once with encode_fsst stubbed to return false; +the columnar-INSERT delta is FSST's ingestion cost, the size delta its ratio. +FSST_EXIT=0 +=== READ STREAM === +== pgColumnar read-stream / AIO benchmark == +PG_CONFIG=/usr/local/pg18_uring/bin/pg_config (PostgreSQL 18.4) +rows=60000000 workdir=/tmp/pgcolumnar-rsbench.wij4kg +-- build + install +-- initdb +-- load 60000000 rows x 8 random bigint columns (columnar, compression=none) + table size: 3675 MB file: base/16384/16514 shared_buffers: 128MB + +=== COLD-SCAN LATENCY: io_method x read_stream (median of 3, ms) === +io_method | rs_on_ms | rs_off_ms | rs_speedup +-----------+--------------+--------------+----------- +sync | 40252.143 | 40237.624 | 1.00x +worker | 39436.133 | 40126.733 | 1.02x +io_uring | 40103.578 | 39850.971 | 0.99x + +=== io_method speedup vs sync (read_stream on) === +sync 40252.143 ms (1.00x vs sync) +worker 39436.133 ms (1.02x vs sync) +io_uring 40103.578 ms (1.00x vs sync) + +== benchmark complete == +RS_EXIT=0 +finished 2026-07-27T13:15:23+00:00 +DONE diff --git a/design/ROADMAP.md b/design/ROADMAP.md index 1d7a5a8..434f0bb 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -48,13 +48,35 @@ export and import, flat and nested, for both Arrow and Parquet, all self-contain (no libarrow/libparquet dependency) and matrix-gated. See [gaps/27-arrow-parquet-interop.md](gaps/27-arrow-parquet-interop.md). -The concrete remaining list is complete as of 2026-07-23. The former item, skip -virtual generated-column storage, is DONE (the flush skips the chunk for -`attgenerated = 'v'` columns and the reader returns their missing value; see -`generated_columns.sh`). Phases E (ALP, FSST, chunk-shared FSST) and F (Z-order -cluster, online compaction/rewrite/recluster, physical page reclaim) also landed -on the native PGCN v1 engine, all matrix-gated on PostgreSQL 15-19. What follows -is Future directions (larger, and some deferred for review). +The concrete remaining list was complete as of 2026-07-23, and a July 2026 +external audit plus the follow-on work has since added and closed another round. +Closed since: `ANALYZE` collecting no column statistics (#154), fetch by row +number being quadratic within a row group (#143), the metadata aggregate losing +its fast path to a single deleted row (#149), imports not maintaining indexes or +enforcing unique and exclusion constraints (#153), and a wide table falling off +the fetch cache into per-row group decode (#157). The audit record is +[EXTERNAL_AUDIT_2026_07.md](EXTERNAL_AUDIT_2026_07.md). + +**Open, in the order they are worth taking:** + +1. **Bulk load throughput** (#155). The write path is about 4.9x slower than heap + on a five-column table and 15x slower than the read path. Measurement moved the + target: there is no per-row call overhead to amortise, since a one-column load + is *faster* than heap, and the cost is per value and additive per column, with + one text column costing more than five integer ones. The varlena write path is + where it lives. Plan in [IMPORT_THROUGHPUT_PLAN.md](IMPORT_THROUGHPUT_PLAN.md). +2. **Deferrable unique constraints on the import path** (#168). Enforced at insert + time rather than deferred to commit: over-strict rather than unsound, and the + fix needs `UNIQUE_CHECK_PARTIAL` plus a queued recheck through the after-trigger + machinery. +3. **`ANALYZE` cost and a point-lookup plan regression** (#171). A point lookup + measured 23.75 ms before statistics existed and 1251.88 ms after, and `ANALYZE` + itself ran for tens of minutes on a wide table in isolation without reproducing + inside the benchmark. Two symptoms, possibly one cause, both arriving with + #159. Autoanalyze runs the expensive half unprompted. +4. **`reltuples` after `ANALYZE`** runs a few percent low, because blocks holding + no row-group data count as visited while offering no rows. The planner does not + use that figure for columnar tables, so this is cosmetic until something does. Deferred (documented, not yet built): end-truncation for lazy disk reclaim (corruption-critical VM-fork/WAL hazards, see PHASE_F_RECLAIM_PLAN.md); the F1 diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 4972364..cc946df 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -17,20 +17,15 @@ Environment variables: `BENCH_SCALE` (rows, default 6000000), `BENCH_REPS` (timed repetitions, median reported, default 5), `BENCH_PORT`, and `BENCH_DUCKDB` (set to 1 to add a DuckDB comparison when `duckdb` is on `PATH`). -The numbers below are one full run of all three harnesses, measured 2026-07-26 at -commit `1be027b`: PostgreSQL 17.10 non-assert (18.4 with io_uring for the read +The numbers below are one full run of all three harnesses, measured 2026-07-27 at +commit `7a9c9f7`: PostgreSQL 17.10 non-assert (18.4 with io_uring for the read stream harness), 6,000,000 rows, 8-column table, median of 5, on 8 cores with 24 GB of memory. Raw output is in -[../bench/sample_output_pg17_6m.txt](../bench/sample_output_pg17_6m.txt). They -show the shape of the tradeoff, not a precise score. The dataset is synthetic and -deliberately mixes column shapes that suit different encodings, so a table of +[../bench/sample_output_all_2026_07_27.txt](../bench/sample_output_all_2026_07_27.txt). +They show the shape of the tradeoff, not a precise score. The dataset is synthetic +and deliberately mixes column shapes that suit different encodings, so a table of purely random values will look worse and a repetitive one better. -One change landed after these were measured and is not reflected in them: #160 -made the zone min/max tracking on the write path 5 to 7% faster, so every -ingestion figure here (loads, imports, the FSST insert) is a floor rather than -what the current tree does. Query latency and storage are unaffected. - **Compare ratios across runs, not absolute milliseconds.** Re-measuring the previous run's commit on this machine on the same day (see [What changed](#what-changed-since-the-previous-run)) reproduced its query @@ -69,20 +64,29 @@ Heap versus columnar (zstd), median milliseconds: | query | heap | columnar | heap / columnar | | --- | --- | --- | --- | -| count(*) full table | 251.34 | 0.02 | 12567 | -| sum/avg over one int column | 354.96 | 0.53 | 670 | -| filtered agg, min/max-skippable range | 263.62 | 92.35 | 2.85 | -| projection: 3 of 8 cols, 1% filter | 264.94 | 81.58 | 3.25 | -| point lookup by indexed id | 0.01 | 23.75 | 0.00 | +| count(*) full table | 254.28 | 0.02 | 12714 | +| sum/avg over one int column | 379.89 | 0.56 | 678 | +| filtered agg, min/max-skippable range | 268.96 | 90.70 | 2.97 | +| projection: 3 of 8 cols, 1% filter | 232.83 | 85.83 | 2.71 | +| point lookup by indexed id | 0.01 | 1251.88 | 0.00 | `count(*)` and the ungrouped aggregates are answered from row-group metadata without decoding column data, which is why they are microseconds rather than milliseconds. -The point lookup is the trade the design makes and does not hide: a single-row -fetch decodes the row group the row lives in, so one row costs a whole group's -decode. Columnar suits scans and aggregates; heap suits point lookups and -write-heavy OLTP. +**The point lookup number is a regression and is under investigation +([issue #171](https://github.com/jdatcmd/pgcolumnar/issues/171)), not a property +of the design.** The previous run recorded 23.75 ms for the same query on the +same machine. Two things are established: it is not the lazy-decoding slot, since +an A/B across that merge on an unanalyzed table gives 16.42 ms before and 13.73 ms +after; and the difference between that probe and this harness is that the harness +runs `ANALYZE` on the table first. So the planner is choosing differently once +statistics exist, and choosing worse. Treat the row as a bug report rather than a +measurement of the fetch path. + +What remains true regardless: a single-row fetch has to locate and decode within +the row group the row lives in, so columnar suits scans and aggregates while heap +suits point lookups and write-heavy OLTP. ## Aggregates fall back once anything is deleted @@ -115,9 +119,9 @@ single run for the delete: | operation | heap | columnar | columnar / heap | | --- | --- | --- | --- | | UPDATE single row by id | 0.02 ms | 0.22 ms | 11 | -| UPDATE 1000 rows, ids in row order | 3.76 ms | 20.78 ms | 5.5 | -| UPDATE 1000 rows, ids scattered | 44.07 ms | 146.07 ms | 3.3 | -| DELETE 1000 rows by id range | 0.5 ms | 22.8 ms | 46 | +| UPDATE 1000 rows, ids in row order | 3.81 ms | 14.28 ms | 3.8 | +| UPDATE 1000 rows, ids scattered | 43.15 ms | 147.89 ms | 3.4 | +| DELETE 1000 rows by id range | 0.5 ms | 14.7 ms | 29 | Row-ordered access does better than scattered because consecutive fetches stay inside one row group, which the statement-scoped decoded-group cache serves @@ -135,14 +139,14 @@ Vectorization on versus off (columnar zstd, median ms): | query | on | off | speedup | | --- | --- | --- | --- | -| sum/avg over int | 0.51 | 1352.10 | 2651 | -| filtered agg (range) | 87.51 | 86.53 | 0.99 | +| sum/avg over int | 0.51 | 1392.60 | 2731 | +| filtered agg (range) | 92.12 | 90.32 | 0.98 | Index-only scan on versus off (covering range count, median ms): | query | on | off | speedup | | --- | --- | --- | --- | -| covering count, id range (~2%) | 7.24 | 689.21 | 95 | +| covering count, id range (~2%) | 7.53 | 698.95 | 93 | The "off" column is the fetch-by-row path doing nothing else, which makes it the clearest single view of what that path costs, and the clearest measure of what @@ -153,18 +157,18 @@ Projection scan on versus off (covering scan on a scattered sort key, median ms) | query | on | off | speedup | | --- | --- | --- | --- | -| sortk, val where sortk in ~0.1% range | 201.16 | 646.43 | 3.21 | +| sortk, val where sortk in ~0.1% range | 191.41 | 635.35 | 3.32 | Sorted storage (`pgcolumnar.vacuum_sorted`), narrow range scan on a key not correlated with insert order, median ms: | state | ms | | --- | --- | -| before vacuum_sorted | 348.79 | -| after vacuum_sorted | 45.47 | +| before vacuum_sorted | 364.13 | +| after vacuum_sorted | 47.72 | Compression none versus zstd (columnar table-only): 40 MB versus 5.95 MB, with -scan latency unchanged (0.50 ms against 0.49 ms), because the encoded stream is +scan latency unchanged (0.52 ms against 0.52 ms), because the encoded stream is already small and the aggregates do not read it. ## Import and export @@ -173,15 +177,15 @@ Export, 6,000,000 rows, 5 columns: | format | ms | file size | M rows/s | | --- | --- | --- | --- | -| arrow | 1029.3 | 186 MB | 5.8 | -| parquet | 1113.7 | 186 MB | 5.4 | +| arrow | 1008.6 | 186 MB | 5.9 | +| parquet | 1100.2 | 186 MB | 5.5 | Import, 6,000,000 rows, 5 columns: | format | ms | M rows/s | | --- | --- | --- | -| arrow | 18533.2 | 0.3 | -| parquet | 18746.0 | 0.3 | +| arrow | 17721.5 | 0.3 | +| parquet | 17762.5 | 0.3 | Import is about 18x slower than export, and the reason is not the import code. Measured separately: `import_arrow` costs 12,150 ms against 12,990 ms for an @@ -204,8 +208,8 @@ column: | format | export ms | import ms | file size | | --- | --- | --- | --- | -| arrow | 580.0 | 4914.0 | 38 MB | -| parquet | 533.4 | 4923.5 | 35 MB | +| arrow | 622.8 | 4904.1 | 38 MB | +| parquet | 533.7 | 4860.1 | 35 MB | Both reconstructed tables matched the source exactly (zero differing rows). @@ -215,8 +219,8 @@ Both reconstructed tables matched the source exactly (zero differing rows). | | | | --- | --- | -| heap INSERT | 1.92 s | -| columnar INSERT | 12.37 s | +| heap INSERT | 2.00 s | +| columnar INSERT | 12.86 s | | heap size | 419 MB | | columnar size | 101 MB | | vectors using FSST | 20 of 20 | @@ -235,9 +239,9 @@ Cold-scan latency on the PostgreSQL 18 io_uring build, 60,000,000 rows, median o | io_method | read stream on | off | gain | | --- | --- | --- | --- | -| `sync` | 38.12 s | 38.05 s | 1.00x | -| `worker` | 37.49 s | 38.24 s | 1.02x | -| `io_uring` | 38.03 s | 38.91 s | 1.02x | +| `sync` | 40.25 s | 40.24 s | 1.00x | +| `worker` | 39.44 s | 40.13 s | 1.02x | +| `io_uring` | 40.10 s | 39.85 s | 0.99x | Across methods with the read stream on: `worker` 1.02x against `sync`, `io_uring` 1.00x. @@ -266,53 +270,43 @@ Reading the Parquet file pgColumnar wrote, 6,000,000 rows, count and sum: | reader | time | | --- | --- | -| DuckDB `read_parquet` (stats-accelerated) | 8 ms | -| pyarrow `read_table` (full materialization) | 128 ms | +| DuckDB `read_parquet` (stats-accelerated) | 12 ms | +| pyarrow `read_table` (full materialization) | 149 ms | These confirm the Parquet output is read by other engines without conversion. ## What changed since the previous run -The previous version of this document recorded a run at commit `2f1320f` and -flagged `count(*)` as slower than it should be, tracked as issue #133. That is -fixed, and two further changes landed while this run was being written up. To -measure honestly rather than compare across machines and days, the same harness -was run on this machine on this day at each commit: - -| metric | 2f1320f | f7adbdb | 1be027b | | -| --- | --- | --- | --- | --- | -| count(*) | 8.27 ms | 0.02 ms | 0.02 ms | 413x | -| sum/avg over int | 8.04 ms | 0.53 ms | 0.53 ms | 15x | -| covering count, index-only scan off | 200914.88 ms | 31845.77 ms | 689.21 ms | 291x | -| DELETE 1000 rows by id range | not measured | 1509.5 ms | 22.8 ms | 66x | -| count(*) with one row deleted | not measured | 222.28 ms | 0.18 ms | 1235x | -| point lookup by id | 32.96 ms | 26.75 ms | 23.75 ms | 1.4x | -| projection: 3 of 8 cols | 91.68 ms | 80.18 ms | 81.58 ms | 1.12x | -| storage, all three tables | identical | identical | identical | no change | - -Four changes account for it: - -- **#133** fixed the aggregate path. The planner had been losing to a parallel - scan on cost, and `count(*)` was reading every column's zone maps and using none - of them. -- **#148** cached the decoded row group for the length of a statement, so - fetching many rows from one group decodes it once. -- **#152** replaced the walk to a row's position with a rank lookup, which is what - takes the index-only-scan-off shape from 31.8 s to 0.69 s and the delete from - 1509 ms to 23 ms. Together with #148 that is issue #143 closed: the path is no - longer quadratic in the rows touched per group. -- **#151** made the delete fallback per row group instead of per storage, so one - deleted row no longer costs the whole table its metadata answers. - -Nothing measured got slower. - -The absolute ingestion and I/O figures in this document are higher than the -previous run recorded (arrow import 18.5 s against 13.6 s, cold scans about 38 s -against about 30 s), and they moved again between the two runs on this same day -(arrow export 923 ms then 1029 ms) without any code touching that path. -Re-running `2f1320f` today reproduces today's figures rather than the recorded -ones, so this is the machine. It is the reason for the warning at the top about -comparing ratios rather than milliseconds. +Measured on the same machine, same harness, at three commits. The middle column +is the run this document previously recorded. + +| metric | 2f1320f | 1be027b | 7a9c9f7 | +| --- | --- | --- | --- | +| count(*) | 8.27 ms | 0.02 ms | 0.02 ms | +| sum/avg over int | 8.04 ms | 0.53 ms | 0.56 ms | +| covering count, index-only scan off | 200,914 ms | 689 ms | 699 ms | +| DELETE 1000 rows by id range | not measured | 22.8 ms | **14.7 ms** | +| UPDATE 1000 rows, ids in row order | not measured | 20.78 ms | **14.28 ms** | +| count(*) with one row deleted | 222.28 ms | 0.18 ms | 0.18 ms | +| point lookup by indexed id | 32.96 ms | 23.75 ms | **1251.88 ms** | +| storage, all three tables | identical | identical | identical | + +The mutation figures improved again, from the direct zone min/max comparison +(#160) and the needed-columns fetch (#164). + +Two things do not appear in this table because they are not in the harness, and +both are larger than anything in it: + +- **A wide table is no longer unusable for index-driven access.** 2,000 index + fetches reading one column of a 41-column table went from 1,001,374 ms to + 614 ms when the lazily-decoding slot landed (#169), and an 11-column table from + 284,148 ms to 159 ms. That is the cliff [issue #157] described, and it is gone. +- **`ANALYZE` now collects statistics** (#159), including correlation, which is + what lets the planner see the locality `vacuum_sorted` and Z-ordering create. + Its cost is unmeasured here and is part of #171. + +The point lookup is the one number that moved the wrong way, and it moved a long +way. See the note above it. ## Reading the results diff --git a/docs/features.md b/docs/features.md index 47c4ac5..63bbb98 100644 --- a/docs/features.md +++ b/docs/features.md @@ -48,6 +48,13 @@ settings see the [configuration reference](configuration.md); for constraints se this: it is answered from each row group's stored row count and reads no column data. Set `pgcolumnar.enable_vectorization` to `off` to force an ordinary aggregate over the scan instead. +- Column statistics: `ANALYZE` samples rows spread across row groups and stores + null fraction, distinct counts, most-common values, histograms and correlation, + so predicates are estimated from the data. Correlation is what lets the planner + see the locality `pgcolumnar.vacuum_sorted` and Z-order clustering create. +- Fetch by row number decodes only the columns the executor asks for and reuses + the decoded row group for the rest of the statement, so an index-driven read of + a wide table does not decode columns it will not return. - Parallel scan across a table's row groups. - Read stream prefetch of block reads on PostgreSQL 17 and later (`pgcolumnar.enable_read_stream`). @@ -125,7 +132,9 @@ coverage. `pgcolumnar.export_parquet(table, path)`, both without a libarrow or libparquet dependency. - Import from Arrow and Parquet: `pgcolumnar.import_arrow(table, path)` and - `pgcolumnar.import_parquet(table, path)` into an existing target table. The + `pgcolumnar.import_parquet(table, path)` into an existing target table The import + maintains every index on the target and enforces unique and exclusion + constraints, so it cannot leave the table in a state ordinary DML would refuse. The Parquet reader parses Thrift metadata, decompresses uncompressed, Snappy, GZIP, ZSTD, and LZ4_RAW pages, and decodes PLAIN and dictionary encodings from data-page versions 1 and 2. diff --git a/docs/limitations.md b/docs/limitations.md index 19f3f06..6f10216 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -26,13 +26,15 @@ only. The rest of the extension runs on any architecture PostgreSQL supports. - Columnar storage is built for append-mostly data. Updates and deletes are supported, but they mark rows rather than rewriting data, and the space is reclaimed only by `pgcolumnar.vacuum`. -- Point lookups are slow relative to heap. A single-row fetch by item pointer must - read and decode the row group that contains the row. Bloom filters speed up an - equality scan by skipping row groups, but do not help an index fetch by item - pointer. -- A bulk `UPDATE` re-fetches each old row by item pointer to fill unchanged - columns, which is proportional to rows times row group size and is not yet - optimized. +- Point lookups are slower than heap, though far less so than they were. A fetch + by item pointer locates the row's group and decodes only the columns the + executor asks for, reusing the decoded group for the rest of the statement, so + the cost no longer scales with the table's width or with the row's position in + its group. Heap still wins a single-row fetch outright. Bloom filters speed up + an equality scan by skipping row groups but do not help a fetch by item pointer. +- Bulk `UPDATE` and `DELETE` reached by index are no longer proportional to rows + times row group size. They still cost several times what heap costs, because + each changed row is marked and rewritten rather than updated in place. ## Planner statistics @@ -52,6 +54,14 @@ blocks that hold no row-group data (the metapage, and space reserved but not yet written) count as visited while offering no rows; the planner does not use that figure for columnar tables. +`ANALYZE` samples rows through the fetch path, so its cost grows with the number +and width of columns rather than only with row count. On a wide table it can take +a long time, and autoanalyze runs it unprompted; see +[issue #171](https://github.com/jdatcmd/pgcolumnar/issues/171), which also covers +a point-lookup plan regression that appears once statistics exist. On a table +where that matters, consider `ALTER TABLE ... ALTER COLUMN ... SET STATISTICS 0` +for columns no predicate uses. + `TABLESAMPLE` is unsupported and says so: it raises an error rather than returning no rows. @@ -115,6 +125,19 @@ for the build, like non-concurrent `CREATE INDEX`. Turn projection scans off wit - `CREATE INDEX CONCURRENTLY` (the concurrent validate path) and partial block-range index builds are not supported. +## Constraints on the import path + +`pgcolumnar.import_arrow` and `pgcolumnar.import_parquet` maintain every index on +the target and enforce unique and exclusion constraints, so an import cannot +reach a state an ordinary `INSERT` would refuse. + +One difference from ordinary DML remains: a **deferrable** unique constraint is +checked as each row is inserted rather than deferred to commit. An import that +would transiently violate uniqueness partway through, and be consistent by the +end, is rejected where ordinary DML would accept it. Enforcing early is +over-strict rather than unsound. Tracked as +[issue #168](https://github.com/jdatcmd/pgcolumnar/issues/168). + ## Vectorized aggregate coverage The vectorized aggregate path covers the single-relation, ungrouped