Phase D6e: native-mode suites green + format-dependent tests and fixes - #65
Merged
Conversation
Runs the full suite in native mode (PGC_NATIVE=1) across PostgreSQL 13-19 and closes the gaps that surfaced. Native is selected by a new instance default GUC so the whole suite can exercise the native path, and the format-layout tests become mode-aware. New mechanism - pgcolumnar.default_format_version GUC (0 = format 2.2, 1 = native). ColumnarTableFormatVersion falls back to it when a table sets no explicit format_version option. This is the single flip point D6f will change; D6e uses it (via PGC_NATIVE in the harness) to run every suite native. - The writer anchors to the format already on disk: a storage that holds 2.2 stripes stays 2.2 and a native storage stays native even after the default flips, so an existing table is never rewritten in a mixed format (ColumnarStorageHasStripes joins ColumnarStorageIsNative for this). Correctness fixes exposed by native-mode runs - Projection scan over a native base returned nothing: the liveness cache was 2.2-only (empty for a native base, so every projected row read as not visible). ColumnarBuildLivenessCache now has a native branch built from row groups and the whole-group row mask. - Concurrent first-inserts to the same native table raced on the storage catalog row (duplicate key on storage_pkey). Creation is now serialized by a transaction advisory lock keyed by the storage id, re-checking against a fresh pushed-active snapshot; a second flush in the same transaction still sees its own row. Pushing the snapshot active satisfies PostgreSQL 18's heap-visibility assertion (registered-or-active). - Parallel scan over a native table returned each row once per participating backend: the native row-group loop ignored the shared parallel counter. It now claims row groups from the counter (columnar_next_group_index), the native counterpart of the 2.2 stripe claim, so each group is read by exactly one worker. Format-dependent tests (mode-aware; native layout is covered by the native_* suites) - lib.sh: stripe_count / chunk_group_count count row groups / vectors in native mode; a native_mode helper and the PGC_NATIVE conf switch. - differential, projections, generated_columns, sorted_projection: the 2.2 catalog-introspection checks (encoding-type codes, bloom presence, min/max columns, chunk-group counts, metadata-count EXPLAIN lines) run against the native catalogs where a clean equivalent exists, and otherwise only in 2.2 mode. The oracle/behaviour checks run in both modes. - corruption, hardening: native-catalog tamper equivalents (poke row_group / column_chunk / bloom) asserting a clean error or safe degradation with the backend surviving. The native reader was verified robust to every case. - native_writer pins the instance default to legacy so its format-selection assertions hold regardless of the harness default. Full PostgreSQL 13-19 matrix, assert-enabled, in both modes: ALL VERSIONS PASSED in default (2.2) mode and in native mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase D6e: native-mode green across PostgreSQL 13-19
Runs the full suite in native mode and closes the gaps it surfaced.
Mechanism
pgcolumnar.default_format_versionGUC (0 = format 2.2, 1 = native).ColumnarTableFormatVersionfalls back to it when a table sets no explicitformat_version. This is the single flip point D6f will change; D6e drives it from the harness (PGC_NATIVE=1) to run every suite native.Correctness fixes exposed by native-mode runs
ColumnarBuildLivenessCachenow has a native branch (row groups + whole-group row mask).storage_pkey— native storage-row creation is now serialized by a transaction advisory lock, re-checking against a fresh pushed-active snapshot (which also satisfies PostgreSQL 18's registered-or-active heap-visibility assertion).Format-dependent tests
Made mode-aware: layout introspection runs against native catalogs where a clean equivalent exists, otherwise only in 2.2 mode (native layout is covered by the
native_*suites).corruption/hardeninggained native-catalog tamper equivalents; the native reader was verified robust to every case.Testing
Full PostgreSQL 13-19 matrix, assert-enabled, in both modes: ALL VERSIONS PASSED in default (2.2) mode and in native mode.
Part of the Phase D6 stacked series. D6f (flip the default + docs) follows.
🤖 Generated with Claude Code