diff --git a/bench/sample_output_all_2026_07_27.txt b/bench/sample_output_all_2026_07_27.txt new file mode 100644 index 0000000..0cac0d2 --- /dev/null +++ b/bench/sample_output_all_2026_07_27.txt @@ -0,0 +1,291 @@ +commit: 7a9c9f7 +=== MAIN SUITE === +started 2026-07-27T12:49:19+00:00 +== pgColumnar benchmark == +PG_CONFIG=/usr/local/pg17_nc/bin/pg_config (PostgreSQL 17.10) +scale=6000000 rows reps=5 workdir=/tmp/pgcolumnar-bench.vqC1qE +-- building (non-assert expected) +-- initdb and start +-- loading 6000000 rows (staging, heap, columnar zstd, columnar none) +-- running queries + +=== TABLE SIZES === + table | total_size | bytes +------------------------+------------+----------- + heap | 707 MB | 741793792 + columnar (zstd) | 135 MB | 141033472 + columnar (none) | 40 MB | 42287104 +(3 rows) + + +(table-only size, excluding indexes) + table | table_size +------------------------+------------ + heap | 579 MB + columnar (none) | 40 MB + columnar (zstd) | 6096 kB +(3 rows) + + +=== QUERY LATENCY: heap vs columnar zstd (median ms of 5) === + id | query | heap_ms | columnar_ms | heap_over_col +----+----------------------------------------+---------+-------------+--------------- + 1 | count(*) full table | 254.28 | 0.02 | 12714.00 + 2 | sum/avg over one int column | 379.89 | 0.56 | 678.38 + 3 | filtered agg, min/max-skippable range | 268.96 | 90.70 | 2.97 + 4 | point lookup by indexed id | 0.01 | 1251.88 | 0.00 + 5 | projection: 3 of 8 cols, 1% filter | 232.83 | 85.83 | 2.71 +(5 rows) + + +=== VECTORIZATION on vs off (columnar zstd, median ms) === + query | on_ms | off_ms | speedup_vec +----------------------+-------+---------+------------- + sum/avg over int | 0.51 | 1392.60 | 2730.59 + filtered agg (range) | 92.12 | 90.32 | 0.98 +(2 rows) + + +=== COMPRESSION none vs zstd (size and scan latency) === + metric | none | zstd | none_over_zstd +------------+-------+---------+---------------- + total size | 40 MB | 6096 kB | 6.77 +(1 row) + + metric | none | zstd | none_over_zstd +-----------------+------+------+---------------- + sum/avg scan ms | 0.52 | 0.52 | + count(*) ms | 0.02 | 0.02 | +(2 rows) + +-- sorted projection +Pager usage is off. +SET +CREATE TABLE +INSERT 0 6000000 +ANALYZE + +=== SORTED PROJECTION: narrow range scan on a scattered key (median ms) === + state | ms +----------------------+-------- + before vacuum_sorted | 364.13 +(1 row) + + vacuum_sorted +--------------- + +(1 row) + +ANALYZE + state | ms +---------------------+------- + after vacuum_sorted | 47.72 +(1 row) + +-- index-only scan +Pager usage is off. +SET +SET +SET +CREATE TABLE +INSERT 0 6000000 +CREATE INDEX +VACUUM +ANALYZE + +=== INDEX-ONLY SCAN on vs off (covering range count, median ms) === + query | on_ms | off_ms | speedup_ios +--------------------------------+-------+--------+------------- + covering count, id range (~2%) | 7.53 | 698.95 | 92.82 +(1 row) + +SET +-- projection scan +Pager usage is off. +SET +CREATE TABLE + add_projection +---------------- + +(1 row) + +INSERT 0 6000000 +ANALYZE + +=== PROJECTION SCAN on vs off (covering row scan on a scattered sort key, median ms) === + query | on_ms | off_ms | speedup_proj +--------------------------------------+--------+--------+-------------- + sortk,val where sortk in ~0.1% range | 191.41 | 635.35 | 3.32 +(1 row) + +SET +-- export +CREATE TABLE +INSERT 0 6000000 +CREATE FUNCTION + +=== EXPORT: Arrow IPC and Parquet (6000000 rows, 5 columns) === + format | ms | file_size | m_rows_per_s +----------+--------+-----------+-------------- + arrow | 1008.6 | 186 MB | 5.9 + parquet | 1100.2 | 186 MB | 5.5 +(2 rows) + +-- import +Pager usage is off. +CREATE TABLE +CREATE TABLE +CREATE FUNCTION + +=== IMPORT: Arrow IPC and Parquet (6000000 rows, 5 columns) === + format | ms | m_rows_per_s +----------+---------+-------------- + arrow | 17721.5 | 0.3 + parquet | 17762.5 | 0.3 +(2 rows) + +(row-count check: expect 6000000 for both) + arrow_rows | parquet_rows +------------+-------------- + 6000000 | 6000000 +(1 row) + +-- nested export/import +Pager usage is off. +NOTICE: type "bnc" does not exist, skipping +DROP TYPE +CREATE TYPE +CREATE TABLE +INSERT 0 1000000 +CREATE TABLE +CREATE TABLE +CREATE FUNCTION + +=== NESTED export + import (1000000 rows: int[3] array + composite) === + format | export_ms | import_ms | file_size +----------+-----------+-----------+----------- + arrow | 622.8 | 4904.1 | 38 MB + parquet | 533.7 | 4860.1 | 35 MB +(2 rows) + +(round-trip check: rows in source but not reconstructed; expect 0/0) + arrow_diff | parquet_diff +------------+-------------- + 0 | 0 +(1 row) + +-- mutation (1000000 rows) + +=== MUTATION: rows reached by index (1000000 rows) === +(median of 5 for UPDATE; DELETE is a single run, since a repeat +would find nothing left to delete) + operation | heap_ms | columnar_ms | columnar_over_heap +------------------------------------+---------+-------------+-------------------- + single row by id | 0.02 | 0.22 | 11.00 + 1000 rows, ids in row order | 3.81 | 14.28 | 3.75 + 1000 rows, ids scattered | 43.15 | 147.89 | 3.43 +(3 rows) + + + operation | heap_ms | columnar_ms | columnar_over_heap +------------------------------------+---------+-------------+-------------------- + DELETE 1000 rows by id range | 0.5 | 14.7 | 29.40 +(1 row) + +(row counts must agree after the mutations) + heap_rows | columnar_rows +-----------+--------------- + 999000 | 999000 +(1 row) + + +=== DuckDB comparison (same data, in-process columnar engine) === +Run Time (s): real 0.330 user 0.983363 sys 0.180577 +┌──────────────┐ +│ count_star() │ +│ int64 │ +├──────────────┤ +│ 6000000 │ +└──────────────┘ +Run Time (s): real 0.001 user 0.000100 sys 0.002808 +┌─────────────┬──────────┐ +│ sum(val) │ avg(val) │ +│ int128 │ double │ +├─────────────┼──────────┤ +│ 29997000000 │ 4999.5 │ +└─────────────┴──────────┘ +Run Time (s): real 0.004 user 0.013057 sys 0.000031 +┌───────────┐ +│ sum(val) │ +│ int128 │ +├───────────┤ +│ 599940000 │ +└───────────┘ +Run Time (s): real 0.001 user 0.001897 sys 0.000000 + +=== Cross-engine read of pgColumnar's Parquet output (same file) === +┌─────────┬─────────────┐ +│ rows │ sum_val │ +│ int64 │ int128 │ +├─────────┼─────────────┤ +│ 6000000 │ 29997000000 │ +└─────────┴─────────────┘ +Run Time (s): real 0.012 user 0.031017 sys 0.006628 +pyarrow read_table: rows=6000000 sum_val=29997000000 149.4 ms + +== benchmark complete == +MAIN_EXIT=0 +=== FSST === +== pgColumnar FSST ingestion micro-benchmark == +PG_CONFIG=/usr/local/pg17_nc/bin/pg_config (PostgreSQL 17.10) +scale=3000000 rows workdir=/tmp/pgcolumnar-fsstbench.WqMqsy +-- building (non-assert expected) +-- initdb and start +-- staging 3000000 rows of shared-substring text +-- timing heap ingestion (baseline) +-- timing columnar (compression=none) ingestion + alter_columnar_table_set +-------------------------- + +(1 row) + +-- verify round-trip (heap vs columnar fingerprint) +-- sizes and FSST usage + +===================== FSST INGESTION GATE ====================== + rows 3000000 + heap INSERT 2003.8 ms + columnar INSERT 12855.6 ms + round-trip MATCH + heap size 439279616 bytes + columnar size 105807872 bytes + url chunks using FSST 20 / 20 (first-vector = FSST) +================================================================ +Run once as shipped and once with encode_fsst stubbed to return false; +the columnar-INSERT delta is FSST's ingestion cost, the size delta its ratio. +FSST_EXIT=0 +=== READ STREAM === +== pgColumnar read-stream / AIO benchmark == +PG_CONFIG=/usr/local/pg18_uring/bin/pg_config (PostgreSQL 18.4) +rows=60000000 workdir=/tmp/pgcolumnar-rsbench.wij4kg +-- build + install +-- initdb +-- load 60000000 rows x 8 random bigint columns (columnar, compression=none) + table size: 3675 MB file: base/16384/16514 shared_buffers: 128MB + +=== COLD-SCAN LATENCY: io_method x read_stream (median of 3, ms) === +io_method | rs_on_ms | rs_off_ms | rs_speedup +-----------+--------------+--------------+----------- +sync | 40252.143 | 40237.624 | 1.00x +worker | 39436.133 | 40126.733 | 1.02x +io_uring | 40103.578 | 39850.971 | 0.99x + +=== io_method speedup vs sync (read_stream on) === +sync 40252.143 ms (1.00x vs sync) +worker 39436.133 ms (1.02x vs sync) +io_uring 40103.578 ms (1.00x vs sync) + +== benchmark complete == +RS_EXIT=0 +finished 2026-07-27T13:15:23+00:00 +DONE diff --git a/design/ROADMAP.md b/design/ROADMAP.md index 1d7a5a8..434f0bb 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -48,13 +48,35 @@ export and import, flat and nested, for both Arrow and Parquet, all self-contain (no libarrow/libparquet dependency) and matrix-gated. See [gaps/27-arrow-parquet-interop.md](gaps/27-arrow-parquet-interop.md). -The concrete remaining list is complete as of 2026-07-23. The former item, skip -virtual generated-column storage, is DONE (the flush skips the chunk for -`attgenerated = 'v'` columns and the reader returns their missing value; see -`generated_columns.sh`). Phases E (ALP, FSST, chunk-shared FSST) and F (Z-order -cluster, online compaction/rewrite/recluster, physical page reclaim) also landed -on the native PGCN v1 engine, all matrix-gated on PostgreSQL 15-19. What follows -is Future directions (larger, and some deferred for review). +The concrete remaining list was complete as of 2026-07-23, and a July 2026 +external audit plus the follow-on work has since added and closed another round. +Closed since: `ANALYZE` collecting no column statistics (#154), fetch by row +number being quadratic within a row group (#143), the metadata aggregate losing +its fast path to a single deleted row (#149), imports not maintaining indexes or +enforcing unique and exclusion constraints (#153), and a wide table falling off +the fetch cache into per-row group decode (#157). The audit record is +[EXTERNAL_AUDIT_2026_07.md](EXTERNAL_AUDIT_2026_07.md). + +**Open, in the order they are worth taking:** + +1. **Bulk load throughput** (#155). The write path is about 4.9x slower than heap + on a five-column table and 15x slower than the read path. Measurement moved the + target: there is no per-row call overhead to amortise, since a one-column load + is *faster* than heap, and the cost is per value and additive per column, with + one text column costing more than five integer ones. The varlena write path is + where it lives. Plan in [IMPORT_THROUGHPUT_PLAN.md](IMPORT_THROUGHPUT_PLAN.md). +2. **Deferrable unique constraints on the import path** (#168). Enforced at insert + time rather than deferred to commit: over-strict rather than unsound, and the + fix needs `UNIQUE_CHECK_PARTIAL` plus a queued recheck through the after-trigger + machinery. +3. **`ANALYZE` cost and a point-lookup plan regression** (#171). A point lookup + measured 23.75 ms before statistics existed and 1251.88 ms after, and `ANALYZE` + itself ran for tens of minutes on a wide table in isolation without reproducing + inside the benchmark. Two symptoms, possibly one cause, both arriving with + #159. Autoanalyze runs the expensive half unprompted. +4. **`reltuples` after `ANALYZE`** runs a few percent low, because blocks holding + no row-group data count as visited while offering no rows. The planner does not + use that figure for columnar tables, so this is cosmetic until something does. Deferred (documented, not yet built): end-truncation for lazy disk reclaim (corruption-critical VM-fork/WAL hazards, see PHASE_F_RECLAIM_PLAN.md); the F1 diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 4972364..cc946df 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -17,20 +17,15 @@ Environment variables: `BENCH_SCALE` (rows, default 6000000), `BENCH_REPS` (timed repetitions, median reported, default 5), `BENCH_PORT`, and `BENCH_DUCKDB` (set to 1 to add a DuckDB comparison when `duckdb` is on `PATH`). -The numbers below are one full run of all three harnesses, measured 2026-07-26 at -commit `1be027b`: PostgreSQL 17.10 non-assert (18.4 with io_uring for the read +The numbers below are one full run of all three harnesses, measured 2026-07-27 at +commit `7a9c9f7`: PostgreSQL 17.10 non-assert (18.4 with io_uring for the read stream harness), 6,000,000 rows, 8-column table, median of 5, on 8 cores with 24 GB of memory. Raw output is in -[../bench/sample_output_pg17_6m.txt](../bench/sample_output_pg17_6m.txt). They -show the shape of the tradeoff, not a precise score. The dataset is synthetic and -deliberately mixes column shapes that suit different encodings, so a table of +[../bench/sample_output_all_2026_07_27.txt](../bench/sample_output_all_2026_07_27.txt). +They show the shape of the tradeoff, not a precise score. The dataset is synthetic +and deliberately mixes column shapes that suit different encodings, so a table of purely random values will look worse and a repetitive one better. -One change landed after these were measured and is not reflected in them: #160 -made the zone min/max tracking on the write path 5 to 7% faster, so every -ingestion figure here (loads, imports, the FSST insert) is a floor rather than -what the current tree does. Query latency and storage are unaffected. - **Compare ratios across runs, not absolute milliseconds.** Re-measuring the previous run's commit on this machine on the same day (see [What changed](#what-changed-since-the-previous-run)) reproduced its query @@ -69,20 +64,29 @@ Heap versus columnar (zstd), median milliseconds: | query | heap | columnar | heap / columnar | | --- | --- | --- | --- | -| count(*) full table | 251.34 | 0.02 | 12567 | -| sum/avg over one int column | 354.96 | 0.53 | 670 | -| filtered agg, min/max-skippable range | 263.62 | 92.35 | 2.85 | -| projection: 3 of 8 cols, 1% filter | 264.94 | 81.58 | 3.25 | -| point lookup by indexed id | 0.01 | 23.75 | 0.00 | +| count(*) full table | 254.28 | 0.02 | 12714 | +| sum/avg over one int column | 379.89 | 0.56 | 678 | +| filtered agg, min/max-skippable range | 268.96 | 90.70 | 2.97 | +| projection: 3 of 8 cols, 1% filter | 232.83 | 85.83 | 2.71 | +| point lookup by indexed id | 0.01 | 1251.88 | 0.00 | `count(*)` and the ungrouped aggregates are answered from row-group metadata without decoding column data, which is why they are microseconds rather than milliseconds. -The point lookup is the trade the design makes and does not hide: a single-row -fetch decodes the row group the row lives in, so one row costs a whole group's -decode. Columnar suits scans and aggregates; heap suits point lookups and -write-heavy OLTP. +**The point lookup number is a regression and is under investigation +([issue #171](https://github.com/jdatcmd/pgcolumnar/issues/171)), not a property +of the design.** The previous run recorded 23.75 ms for the same query on the +same machine. Two things are established: it is not the lazy-decoding slot, since +an A/B across that merge on an unanalyzed table gives 16.42 ms before and 13.73 ms +after; and the difference between that probe and this harness is that the harness +runs `ANALYZE` on the table first. So the planner is choosing differently once +statistics exist, and choosing worse. Treat the row as a bug report rather than a +measurement of the fetch path. + +What remains true regardless: a single-row fetch has to locate and decode within +the row group the row lives in, so columnar suits scans and aggregates while heap +suits point lookups and write-heavy OLTP. ## Aggregates fall back once anything is deleted @@ -115,9 +119,9 @@ single run for the delete: | operation | heap | columnar | columnar / heap | | --- | --- | --- | --- | | UPDATE single row by id | 0.02 ms | 0.22 ms | 11 | -| UPDATE 1000 rows, ids in row order | 3.76 ms | 20.78 ms | 5.5 | -| UPDATE 1000 rows, ids scattered | 44.07 ms | 146.07 ms | 3.3 | -| DELETE 1000 rows by id range | 0.5 ms | 22.8 ms | 46 | +| UPDATE 1000 rows, ids in row order | 3.81 ms | 14.28 ms | 3.8 | +| UPDATE 1000 rows, ids scattered | 43.15 ms | 147.89 ms | 3.4 | +| DELETE 1000 rows by id range | 0.5 ms | 14.7 ms | 29 | Row-ordered access does better than scattered because consecutive fetches stay inside one row group, which the statement-scoped decoded-group cache serves @@ -135,14 +139,14 @@ Vectorization on versus off (columnar zstd, median ms): | query | on | off | speedup | | --- | --- | --- | --- | -| sum/avg over int | 0.51 | 1352.10 | 2651 | -| filtered agg (range) | 87.51 | 86.53 | 0.99 | +| sum/avg over int | 0.51 | 1392.60 | 2731 | +| filtered agg (range) | 92.12 | 90.32 | 0.98 | Index-only scan on versus off (covering range count, median ms): | query | on | off | speedup | | --- | --- | --- | --- | -| covering count, id range (~2%) | 7.24 | 689.21 | 95 | +| covering count, id range (~2%) | 7.53 | 698.95 | 93 | The "off" column is the fetch-by-row path doing nothing else, which makes it the clearest single view of what that path costs, and the clearest measure of what @@ -153,18 +157,18 @@ Projection scan on versus off (covering scan on a scattered sort key, median ms) | query | on | off | speedup | | --- | --- | --- | --- | -| sortk, val where sortk in ~0.1% range | 201.16 | 646.43 | 3.21 | +| sortk, val where sortk in ~0.1% range | 191.41 | 635.35 | 3.32 | Sorted storage (`pgcolumnar.vacuum_sorted`), narrow range scan on a key not correlated with insert order, median ms: | state | ms | | --- | --- | -| before vacuum_sorted | 348.79 | -| after vacuum_sorted | 45.47 | +| before vacuum_sorted | 364.13 | +| after vacuum_sorted | 47.72 | Compression none versus zstd (columnar table-only): 40 MB versus 5.95 MB, with -scan latency unchanged (0.50 ms against 0.49 ms), because the encoded stream is +scan latency unchanged (0.52 ms against 0.52 ms), because the encoded stream is already small and the aggregates do not read it. ## Import and export @@ -173,15 +177,15 @@ Export, 6,000,000 rows, 5 columns: | format | ms | file size | M rows/s | | --- | --- | --- | --- | -| arrow | 1029.3 | 186 MB | 5.8 | -| parquet | 1113.7 | 186 MB | 5.4 | +| arrow | 1008.6 | 186 MB | 5.9 | +| parquet | 1100.2 | 186 MB | 5.5 | Import, 6,000,000 rows, 5 columns: | format | ms | M rows/s | | --- | --- | --- | -| arrow | 18533.2 | 0.3 | -| parquet | 18746.0 | 0.3 | +| arrow | 17721.5 | 0.3 | +| parquet | 17762.5 | 0.3 | Import is about 18x slower than export, and the reason is not the import code. Measured separately: `import_arrow` costs 12,150 ms against 12,990 ms for an @@ -204,8 +208,8 @@ column: | format | export ms | import ms | file size | | --- | --- | --- | --- | -| arrow | 580.0 | 4914.0 | 38 MB | -| parquet | 533.4 | 4923.5 | 35 MB | +| arrow | 622.8 | 4904.1 | 38 MB | +| parquet | 533.7 | 4860.1 | 35 MB | Both reconstructed tables matched the source exactly (zero differing rows). @@ -215,8 +219,8 @@ Both reconstructed tables matched the source exactly (zero differing rows). | | | | --- | --- | -| heap INSERT | 1.92 s | -| columnar INSERT | 12.37 s | +| heap INSERT | 2.00 s | +| columnar INSERT | 12.86 s | | heap size | 419 MB | | columnar size | 101 MB | | vectors using FSST | 20 of 20 | @@ -235,9 +239,9 @@ Cold-scan latency on the PostgreSQL 18 io_uring build, 60,000,000 rows, median o | io_method | read stream on | off | gain | | --- | --- | --- | --- | -| `sync` | 38.12 s | 38.05 s | 1.00x | -| `worker` | 37.49 s | 38.24 s | 1.02x | -| `io_uring` | 38.03 s | 38.91 s | 1.02x | +| `sync` | 40.25 s | 40.24 s | 1.00x | +| `worker` | 39.44 s | 40.13 s | 1.02x | +| `io_uring` | 40.10 s | 39.85 s | 0.99x | Across methods with the read stream on: `worker` 1.02x against `sync`, `io_uring` 1.00x. @@ -266,53 +270,43 @@ Reading the Parquet file pgColumnar wrote, 6,000,000 rows, count and sum: | reader | time | | --- | --- | -| DuckDB `read_parquet` (stats-accelerated) | 8 ms | -| pyarrow `read_table` (full materialization) | 128 ms | +| DuckDB `read_parquet` (stats-accelerated) | 12 ms | +| pyarrow `read_table` (full materialization) | 149 ms | These confirm the Parquet output is read by other engines without conversion. ## What changed since the previous run -The previous version of this document recorded a run at commit `2f1320f` and -flagged `count(*)` as slower than it should be, tracked as issue #133. That is -fixed, and two further changes landed while this run was being written up. To -measure honestly rather than compare across machines and days, the same harness -was run on this machine on this day at each commit: - -| metric | 2f1320f | f7adbdb | 1be027b | | -| --- | --- | --- | --- | --- | -| count(*) | 8.27 ms | 0.02 ms | 0.02 ms | 413x | -| sum/avg over int | 8.04 ms | 0.53 ms | 0.53 ms | 15x | -| covering count, index-only scan off | 200914.88 ms | 31845.77 ms | 689.21 ms | 291x | -| DELETE 1000 rows by id range | not measured | 1509.5 ms | 22.8 ms | 66x | -| count(*) with one row deleted | not measured | 222.28 ms | 0.18 ms | 1235x | -| point lookup by id | 32.96 ms | 26.75 ms | 23.75 ms | 1.4x | -| projection: 3 of 8 cols | 91.68 ms | 80.18 ms | 81.58 ms | 1.12x | -| storage, all three tables | identical | identical | identical | no change | - -Four changes account for it: - -- **#133** fixed the aggregate path. The planner had been losing to a parallel - scan on cost, and `count(*)` was reading every column's zone maps and using none - of them. -- **#148** cached the decoded row group for the length of a statement, so - fetching many rows from one group decodes it once. -- **#152** replaced the walk to a row's position with a rank lookup, which is what - takes the index-only-scan-off shape from 31.8 s to 0.69 s and the delete from - 1509 ms to 23 ms. Together with #148 that is issue #143 closed: the path is no - longer quadratic in the rows touched per group. -- **#151** made the delete fallback per row group instead of per storage, so one - deleted row no longer costs the whole table its metadata answers. - -Nothing measured got slower. - -The absolute ingestion and I/O figures in this document are higher than the -previous run recorded (arrow import 18.5 s against 13.6 s, cold scans about 38 s -against about 30 s), and they moved again between the two runs on this same day -(arrow export 923 ms then 1029 ms) without any code touching that path. -Re-running `2f1320f` today reproduces today's figures rather than the recorded -ones, so this is the machine. It is the reason for the warning at the top about -comparing ratios rather than milliseconds. +Measured on the same machine, same harness, at three commits. The middle column +is the run this document previously recorded. + +| metric | 2f1320f | 1be027b | 7a9c9f7 | +| --- | --- | --- | --- | +| count(*) | 8.27 ms | 0.02 ms | 0.02 ms | +| sum/avg over int | 8.04 ms | 0.53 ms | 0.56 ms | +| covering count, index-only scan off | 200,914 ms | 689 ms | 699 ms | +| DELETE 1000 rows by id range | not measured | 22.8 ms | **14.7 ms** | +| UPDATE 1000 rows, ids in row order | not measured | 20.78 ms | **14.28 ms** | +| count(*) with one row deleted | 222.28 ms | 0.18 ms | 0.18 ms | +| point lookup by indexed id | 32.96 ms | 23.75 ms | **1251.88 ms** | +| storage, all three tables | identical | identical | identical | + +The mutation figures improved again, from the direct zone min/max comparison +(#160) and the needed-columns fetch (#164). + +Two things do not appear in this table because they are not in the harness, and +both are larger than anything in it: + +- **A wide table is no longer unusable for index-driven access.** 2,000 index + fetches reading one column of a 41-column table went from 1,001,374 ms to + 614 ms when the lazily-decoding slot landed (#169), and an 11-column table from + 284,148 ms to 159 ms. That is the cliff [issue #157] described, and it is gone. +- **`ANALYZE` now collects statistics** (#159), including correlation, which is + what lets the planner see the locality `vacuum_sorted` and Z-ordering create. + Its cost is unmeasured here and is part of #171. + +The point lookup is the one number that moved the wrong way, and it moved a long +way. See the note above it. ## Reading the results diff --git a/docs/features.md b/docs/features.md index 47c4ac5..63bbb98 100644 --- a/docs/features.md +++ b/docs/features.md @@ -48,6 +48,13 @@ settings see the [configuration reference](configuration.md); for constraints se this: it is answered from each row group's stored row count and reads no column data. Set `pgcolumnar.enable_vectorization` to `off` to force an ordinary aggregate over the scan instead. +- Column statistics: `ANALYZE` samples rows spread across row groups and stores + null fraction, distinct counts, most-common values, histograms and correlation, + so predicates are estimated from the data. Correlation is what lets the planner + see the locality `pgcolumnar.vacuum_sorted` and Z-order clustering create. +- Fetch by row number decodes only the columns the executor asks for and reuses + the decoded row group for the rest of the statement, so an index-driven read of + a wide table does not decode columns it will not return. - Parallel scan across a table's row groups. - Read stream prefetch of block reads on PostgreSQL 17 and later (`pgcolumnar.enable_read_stream`). @@ -125,7 +132,9 @@ coverage. `pgcolumnar.export_parquet(table, path)`, both without a libarrow or libparquet dependency. - Import from Arrow and Parquet: `pgcolumnar.import_arrow(table, path)` and - `pgcolumnar.import_parquet(table, path)` into an existing target table. The + `pgcolumnar.import_parquet(table, path)` into an existing target table The import + maintains every index on the target and enforces unique and exclusion + constraints, so it cannot leave the table in a state ordinary DML would refuse. The Parquet reader parses Thrift metadata, decompresses uncompressed, Snappy, GZIP, ZSTD, and LZ4_RAW pages, and decodes PLAIN and dictionary encodings from data-page versions 1 and 2. diff --git a/docs/limitations.md b/docs/limitations.md index 19f3f06..6f10216 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -26,13 +26,15 @@ only. The rest of the extension runs on any architecture PostgreSQL supports. - Columnar storage is built for append-mostly data. Updates and deletes are supported, but they mark rows rather than rewriting data, and the space is reclaimed only by `pgcolumnar.vacuum`. -- Point lookups are slow relative to heap. A single-row fetch by item pointer must - read and decode the row group that contains the row. Bloom filters speed up an - equality scan by skipping row groups, but do not help an index fetch by item - pointer. -- A bulk `UPDATE` re-fetches each old row by item pointer to fill unchanged - columns, which is proportional to rows times row group size and is not yet - optimized. +- Point lookups are slower than heap, though far less so than they were. A fetch + by item pointer locates the row's group and decodes only the columns the + executor asks for, reusing the decoded group for the rest of the statement, so + the cost no longer scales with the table's width or with the row's position in + its group. Heap still wins a single-row fetch outright. Bloom filters speed up + an equality scan by skipping row groups but do not help a fetch by item pointer. +- Bulk `UPDATE` and `DELETE` reached by index are no longer proportional to rows + times row group size. They still cost several times what heap costs, because + each changed row is marked and rewritten rather than updated in place. ## Planner statistics @@ -52,6 +54,14 @@ blocks that hold no row-group data (the metapage, and space reserved but not yet written) count as visited while offering no rows; the planner does not use that figure for columnar tables. +`ANALYZE` samples rows through the fetch path, so its cost grows with the number +and width of columns rather than only with row count. On a wide table it can take +a long time, and autoanalyze runs it unprompted; see +[issue #171](https://github.com/jdatcmd/pgcolumnar/issues/171), which also covers +a point-lookup plan regression that appears once statistics exist. On a table +where that matters, consider `ALTER TABLE ... ALTER COLUMN ... SET STATISTICS 0` +for columns no predicate uses. + `TABLESAMPLE` is unsupported and says so: it raises an error rather than returning no rows. @@ -115,6 +125,19 @@ for the build, like non-concurrent `CREATE INDEX`. Turn projection scans off wit - `CREATE INDEX CONCURRENTLY` (the concurrent validate path) and partial block-range index builds are not supported. +## Constraints on the import path + +`pgcolumnar.import_arrow` and `pgcolumnar.import_parquet` maintain every index on +the target and enforce unique and exclusion constraints, so an import cannot +reach a state an ordinary `INSERT` would refuse. + +One difference from ordinary DML remains: a **deferrable** unique constraint is +checked as each row is inserted rather than deferred to commit. An import that +would transiently violate uniqueness partway through, and be consistent by the +end, is rejected where ordinary DML would accept it. Enforcing early is +over-strict rather than unsound. Tracked as +[issue #168](https://github.com/jdatcmd/pgcolumnar/issues/168). + ## Vectorized aggregate coverage The vectorized aggregate path covers the single-relation, ungrouped