docs(delta-harness): document standard matrix - #707
Draft
mkuchenbecker wants to merge 17 commits into
Draft
Conversation
Adds a self-contained Scala behavioral test harness that characterizes OpenHouse + Apache Iceberg table behavior end-to-end. The harness crosses a large matrix of table layouts (partitioning, MoR/CoW, ordered writes, nested types) with DDL, DML, maintenance, branching/WAP, streaming, and negative-path operations, asserting deltas against observed pre-state so each case holds under any layout. It runs locally against a real embedded OpenHouse catalog (harness/openhouse/Env.scala boots OpenHouseLocalServer + the OpenHouse Spark catalog; see run-openhouse.sh and HARNESS-GUIDE.md). The scenario and framework sources are also structured as a publishable Gradle library module (openhouse-spark-delta-harness_2.12) that excludes the embedded-only Env so downstream environments can supply their own adapter. Genuine product or upstream bugs are tagged in Plan.knownBugs with a prose explanation and skipped rather than silently passed, so the suite stays green while documenting the defect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds TESTING-MATRIX.md, a living reference that explains the harness as a cross product of independent axes (operation family, data file format, partitioning, write mode, schema, preparation lineage, and reference routing). Documents how a case id reads, the CoreTable/NestedTypesTable/TypesTable schemas, the table layouts, the preparation lineages, and each operation family including the DDL sub-families. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move each test's preparation, action, and assertions into its scenario file so the complete behavior is readable in one place. Keep reusable preparation recipes while creating a fresh table for every case. Preserve the exact 2,574-case catalog, ordering, and known-bug behavior with regression tests for the catalog fingerprint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Explain the scenario-owned test structure, immutable preparations, and fresh-table isolation used by the localized test cases. Describe the matrix as living documentation for the current harness architecture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the harness guide and testing matrix from the implementation PR so the documentation can be reviewed in a separate stacked change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Separate table preparations from DML operations so each case shows its starting state, mutation, and relative assertions in one place. Keep feature-owned scenarios in removable RTAS, merge-on-read, and branch layers while preserving the exact ordered 2,572-case catalog. Run the same published sources through the local Gradle task and the acceptance adapter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the standard branch focused on copy-on-write behavior, shared table preparations, bespoke DDL coverage, and the local execution framework. Remove RTAS, merge-on-read, branch, and WAP scenario ownership from this layer. Pin the resulting ordered standard catalog at 1,181 cases so each child branch can add one reviewable feature delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep runtime case metadata limited to stable identifiers and execution state. Put preparation and test explanations beside their Scala behavior so reviewers can read each case without tracing string registries. Generate a fresh UUID for every table and begin cleanup only after the preparation creates it, which preserves any pre-existing table on a name conflict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract the owned-table cleanup state machine behind a package-private boundary so its failure paths can be tested without starting Spark. Pin conflict preservation, successful cleanup, and suppression of cleanup failure behind the primary test failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pin the remaining ownership outcome: when the test body succeeds and cleanup fails, the cleanup failure must surface to the runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Sep 1, 2026
Reflow harness documentation to the repository's 120-column target and explain the DML operation and preparation matrix at its source. Name the reusable date column independently from partitioning so layouts, not column identifiers, express partition choices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace provenance and consequence buckets with capability-owned scenario files whose public contribution surfaces explain the catalog at a glance. Separate local runner code from the publishable harness, make preparations show creation and standard seeding explicitly, and reindex generic case IDs. Use generated table names and failure-preserving ownership boundaries for every case-owned table, view, registration, rename, and lock lifecycle. Move column-default coverage out of the standard layer for a dedicated follow-up PR while pinning the remaining 1,177-case catalog. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Name every scenario source and trait ScenarioFoo so scenario files group together and the framework files remain visually distinct. Preserve the catalog contributions, IDs, ordering, count, and fingerprint unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep PR 682 focused on reusable DDL and DML coverage while moving orthogonal capabilities to extension branches. - retain 642 Parquet and ORC foundation cases - extract reusable changelog and concurrency support - preserve Plan and Scenarios consumer compatibility - add extension-stable catalog and support contract tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 264 replacement-specific cases on the 642-case foundation. - run every reusable DML operation across four RTAS preparations - cover schema, partition, policy, history, rename, order, and identity - require precise lineage rejection and typed concurrency outcomes - expose silent narrowing corruption as a known product bug Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the replace-versus-append assertion limited to serializable outcomes while recording the rare lost-replace result as a known bug. Both format cases remain available for re-enablement when concurrent RTAS commits report conflicts or preserve the replace. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the shared local and acceptance execution model, lifecycle ownership, catalog composition, coverage matrix, and extension graph. Pin the final branch counts and fingerprints so future changes must state their intended test-surface impact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mkuchenbecker
force-pushed
the
mkuchenbecker/delta-harness-docs
branch
from
September 2, 2026 09:47
f2833ab to
f6edb4f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR documents the standard delta-harness layer from PR 682. It explains how source-attached preparation and test documentation, relative assertions, bespoke cases, fresh-table ownership, and the local Gradle runner fit together.
The source branch is based directly on the standard framework. Because this PR targets
main, its current diff also includes PR 682 until that dependency merges. RTAS, merge-on-read, branch, and WAP implementation details remain in the code PRs.Stack
All OpenHouse PRs target
main. Review them in dependency order; each upper diff shrinks as its dependencies merge. The documentation and RTAS PRs both depend on the standard framework, and the Airflow PR consumes the branch and WAP artifact.mainDocuments
HARNESS-GUIDE.mdTESTING-MATRIX.mdValidation
HARNESS-GUIDE.mdandTESTING-MATRIX.md.