test(delta-harness): add standard matrix by mkuchenbecker · Pull Request #682 · linkedin/openhouse

mkuchenbecker · 2026-08-13T20:08:33Z

Summary

This PR adds the standard OpenHouse delta-harness foundation. Preparations document the table state they create, test definitions document the operation and observable result beside their assertions, and runtime output uses stable case IDs.

The standard catalog contains 1,181 cases. Replace Table As Select (RTAS), merge-on-read, branch, and write-audit-publish (WAP) coverage are isolated in stacked child PRs.

Stack

All OpenHouse PRs target main. Review them in dependency order; each upper diff shrinks as its dependencies merge. The documentation PR depends on the standard framework, and the Airflow PR consumes the branch and WAP artifact.

Order	Layer	Review	Depends on	Cases
1	Standard framework	#682 (this PR)	`main`	1,181
2	Replace Table As Select (RTAS)	#704	#682	1,542
3	Merge-on-read	#705	#704	2,008
4	Branch and write-audit-publish (WAP)	#706	#705	2,572
Side	Standard documentation	#707	#682	Standard only
Downstream	Airflow acceptance	#2266	Branch and WAP artifact	2,572

Scope

Area	Change
Preparations	Name each reusable starting state and put Scaladoc describing its layout, rows, lineage, properties, or schema beside the definition.
DML tests	Define 54 reusable operations with the action and assertions together. Mutations assert complete row sets and snapshot deltas relative to the prepared state.
Bespoke tests	Keep DDL, maintenance, reader/writer, interaction, surface, and hazard setup with the operation and assertions it serves.
Runtime contract	Keep `Plan.Case`, `TablePreparation`, and `DmlTestCase` limited to stable IDs, executable behavior, and skip or known-bug metadata.
Execution	Run the same scenario source locally through Gradle and publish it as a portable Scala module for downstream acceptance tests.
Isolation	Generate every table name from a fresh UUID plus the existing counter. Cleanup starts only after creation succeeds and drops only the table owned by that case.

Validation

The catalog fingerprint is the regression guard for case identity and order.

Check	Result
Module tests	Passed on JDK 17.
Catalog	1,181 unique cases with SHA-256 `377f65959e3034c51e078fea72491444b06a6055f37c051184bdc379234b3d57`.
Table lifecycle	Tests cover UUID uniqueness, cleanup after successful ownership, conflict preservation, and cleanup-failure propagation.
Representative DML	`read.projection` passed locally.
Representative DDL	`ddl.renameTable` passed locally.
Review	Writing, pedantic, architecture, and testing reviews have no remaining findings.

Run a local slice with:

export JAVA_HOME=$(/usr/libexec/java_home -v 17)
./gradlew --no-daemon \
  :integrations:spark:openhouse-spark-delta-harness_2.12:runOpenHouse \
  --args='read.projection'

Adds a self-contained Scala behavioral test harness that characterizes OpenHouse + Apache Iceberg table behavior end-to-end. The harness crosses a large matrix of table layouts (partitioning, MoR/CoW, ordered writes, nested types) with DDL, DML, maintenance, branching/WAP, streaming, and negative-path operations, asserting deltas against observed pre-state so each case holds under any layout. It runs locally against a real embedded OpenHouse catalog (harness/openhouse/Env.scala boots OpenHouseLocalServer + the OpenHouse Spark catalog; see run-openhouse.sh and HARNESS-GUIDE.md). The scenario and framework sources are also structured as a publishable Gradle library module (openhouse-spark-delta-harness_2.12) that excludes the embedded-only Env so downstream environments can supply their own adapter. Genuine product or upstream bugs are tagged in Plan.knownBugs with a prose explanation and skipped rather than silently passed, so the suite stays green while documenting the defect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds TESTING-MATRIX.md, a living reference that explains the harness as a cross product of independent axes (operation family, data file format, partitioning, write mode, schema, preparation lineage, and reference routing). Documents how a case id reads, the CoreTable/NestedTypesTable/TypesTable schemas, the table layouts, the preparation lineages, and each operation family including the DDL sub-families. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move each test's preparation, action, and assertions into its scenario file so the complete behavior is readable in one place. Keep reusable preparation recipes while creating a fresh table for every case. Preserve the exact 2,574-case catalog, ordering, and known-bug behavior with regression tests for the catalog fingerprint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Explain the scenario-owned test structure, immutable preparations, and fresh-table isolation used by the localized test cases. Describe the matrix as living documentation for the current harness architecture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove the harness guide and testing matrix from the implementation PR so the documentation can be reviewed in a separate stacked change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-08-26T16:20:56Z

+          TablePreparation(
+            layout.label,
+            createAndSeed(layout, 3)
+              .sql("ddl")(table => s"ALTER TABLE $table ADD COLUMN cc int")(),


what ddl does is unclear here. ITs a function.

mkuchenbecker · 2026-08-26T16:21:21Z

+          layout.label.endsWith("/orc"))
+      .flatMap { layout =>
+        val preparations = List(
+          TablePreparation(


Each preparation needs a description.

mkuchenbecker · 2026-08-26T16:21:46Z

+        layout.label.endsWith("/parquet") ||
+          layout.label.endsWith("/orc"))


Why not just ahve a list of layouts?

mkuchenbecker · 2026-08-26T16:22:15Z

+            "ddlConsume:writeOrder."),
+          TablePreparation(
+            layout.label,
+            createAndSeed(layout, 3)


Create and seed should be separate calls. What is seeded should be visible or standard.

Separate table preparations from DML operations so each case shows its starting state, mutation, and relative assertions in one place. Keep feature-owned scenarios in removable RTAS, merge-on-read, and branch layers while preserving the exact ordered 2,572-case catalog. Run the same published sources through the local Gradle task and the acceptance adapter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep the standard branch focused on copy-on-write behavior, shared table preparations, bespoke DDL coverage, and the local execution framework. Remove RTAS, merge-on-read, branch, and WAP scenario ownership from this layer. Pin the resulting ordered standard catalog at 1,181 cases so each child branch can add one reviewable feature delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-08-26T23:05:10Z

+    // (b) toJson takes no format-version argument, so the key serializes the same regardless of format version.
+    // (c) The value round-trips through fromJson then toJson.
+    val reparsed = org.apache.iceberg.SchemaParser.fromJson(json)
+    val json2 = org.apache.iceberg.SchemaParser.toJson(reparsed)


Just import. Apply this across all of these deep calls.

unaddressed

Keep runtime case metadata limited to stable identifiers and execution state. Put preparation and test explanations beside their Scala behavior so reviewers can read each case without tracing string registries. Generate a fresh UUID for every table and begin cleanup only after the preparation creates it, which preserves any pre-existing table on a name conflict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Extract the owned-table cleanup state machine behind a package-private boundary so its failure paths can be tested without starting Spark. Pin conflict preservation, successful cleanup, and suppression of cleanup failure behind the primary test failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pin the remaining ownership outcome: when the test body succeeds and cleanup fails, the cleanup failure must surface to the runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-09-01T19:09:45Z

+    deleteByNullCondition)
+
+  /**
+   * DELETE WHERE datepartition = '2024-01-01-00' removes the rows in that partition value, keeps


datepartition is a smell. Instead of that, its a date column that is partitioned. Adding the name partition to the col sows confusion.

mkuchenbecker · 2026-09-01T19:13:57Z

+// size, and the compaction plan. These behaviors have no catalog SQL surface of their own, so a
+// case reaches them through the Iceberg API or a Spark configuration and asserts the result a
+// caller can observe.
+trait ForkScenarios extends ScenarioKit {


forkscenarios are Iceberg tests. they should all run agains the fork, so its unclear why this is a file vs denormalized into other features.

mkuchenbecker · 2026-09-01T19:14:53Z

+   * Reflectively builds an optional int NestedField carrying the given initial default. Returns
+   * None when the builder API is absent, so a caller can assert that absence directly.
+   */
+  private def buildDefaultedIntField(id: Int, name: String, dflt: Int): Option[org.apache.iceberg.types.Types.NestedField] = {


move default values to its own PR.

mkuchenbecker · 2026-09-01T19:16:34Z

+   * DataFrame into a 4-partition table under each mode therefore yields at least as many files
+   * under the default as under HASH. The file format is the parameter.
+   */
+  private def forkPartitionDistDefault(fmt: String)(ctx: Ctx): Unit = {


this test doesn't seem useful. is it? a write distribution mode test makes sense but not a None test. To me this is purely spark and nothing specific to iceberg being tested. That may be fine depending on how we phrase it.

mkuchenbecker · 2026-09-01T19:17:03Z

+   * path that supplies one. Reflection reaches the builder and getProperties because some Iceberg
+   * artifacts leave them out of the public compiled API.
+   */
+  private def forkFileReplicationFactor(ctx: Ctx): Unit = {


REplicaiton factor should be tested. its not a fork, but a feature.

mkuchenbecker · 2026-09-01T19:17:27Z

+   * plans one task group, and a split size below one file plans one group per file. The file format
+   * is the parameter.
+   */
+  private def forkSplitSize(fmt: String)(ctx: Ctx): Unit = {


again, these are iceberg features not a "fork"

mkuchenbecker · 2026-09-01T19:18:22Z

+import scala.reflect.{ClassTag, classTag}
+import scala.util.control.NonFatal
+
+// The copy-on-write reader, writer and hazard families. The reader and writer cases pin the


CoW vs MoR is the test. "hazard" makes no sense.

mkuchenbecker

I would reindex on capabilities vs Hazard / Fork / etc. More files is fine, ideal case is a new file + a delta to integrate it.

mkuchenbecker · 2026-09-01T19:22:35Z

+  val typesLayouts: List[Layout] =
+    List("parquet", "orc", "avro").map(format => Layout(s"types-unpartitioned/$format", table =>


constructing this list literal is suspect vs getting a standard "AllFileTypes". this is a nit for now, but a standard list is perferred.

Reflow harness documentation to the repository's 120-column target and explain the DML operation and preparation matrix at its source. Name the reusable date column independently from partitioning so layouts, not column identifiers, express partition choices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-09-01T19:55:02Z

+object Main {
+  def main(args: Array[String]): Unit = {


Main seems to ahve nothing to do with env.

mkuchenbecker · 2026-09-01T19:57:13Z

+    }
+
+  /** The rejected DML statements, on the preparedCoreFormats preparations. */
+  val negativeCases: List[Plan.Case] =


compositions seem like they should be at teh top or bottom (top is preferred IMO). We can organize by public and private.

Replace provenance and consequence buckets with capability-owned scenario files whose public contribution surfaces explain the catalog at a glance. Separate local runner code from the publishable harness, make preparations show creation and standard seeding explicitly, and reindex generic case IDs. Use generated table names and failure-preserving ownership boundaries for every case-owned table, view, registration, rename, and lock lifecycle. Move column-default coverage out of the standard layer for a dedicated follow-up PR while pinning the remaining 1,177-case catalog. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-09-02T00:58:05Z

@@ -0,0 +1,219 @@
+package harness


This should fail when you attempt to do so across RTAS to a previous table. Make sure RTAS PR capturs that.

mkuchenbecker · 2026-09-02T00:58:49Z

@@ -0,0 +1,43 @@
+package harness
+
+/**


Stack on top unless tags need RTAS testing.

mkuchenbecker · 2026-09-02T00:59:12Z

@@ -0,0 +1,137 @@
+package harness


can be stacked on top

mkuchenbecker · 2026-09-02T00:59:46Z

+import java.nio.file.{Files, Paths}
+
+/**
+ * Encryption: the OSS build writes table data in plaintext, because OpenHouse delegates table-data encryption to an


can be stacked on top unless there is the chance RTAS silently drops encryption.

mkuchenbecker · 2026-09-02T01:01:15Z

+
+  /** The plaintext data-file case, on the standard seeded Parquet table. */
+  lazy val encryptionCases: List[Plan.Case] =
+    List(dataFilePlaintextCase(preparedStandardTable("parquet")))


orc should be the default test choice or both orc and parquet. Both is better. I would jsut standardize to do both.

mkuchenbecker · 2026-09-02T01:05:59Z

@@ -0,0 +1,61 @@
+package harness


rtas should test that after an RTAS sort order can be changed or dropped if its not already

mkuchenbecker · 2026-09-02T01:06:33Z

@@ -0,0 +1,213 @@
+package harness


can be stacked on top

mkuchenbecker · 2026-09-02T01:07:04Z

@@ -0,0 +1,98 @@
+package harness


RTAS definately needs time travel testing as a negative test case.

mkuchenbecker · 2026-09-02T01:07:24Z

@@ -0,0 +1,161 @@
+package harness


can be stacked on top

mkuchenbecker · 2026-09-02T01:07:37Z

@@ -0,0 +1,50 @@
+package harness


can be stacked on top

Name every scenario source and trait ScenarioFoo so scenario files group together and the framework files remain visually distinct. Preserve the catalog contributions, IDs, ordering, count, and fingerprint unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-09-02T01:37:51Z

+   * here rather than derived from `Plan.contributions`, so adding, dropping, renaming or reordering a capability fails
+   * this test until the intended catalog shape is restated.
+   */
+  private val expectedContributionNames = List(


this test is a tautaulogy.

mkuchenbecker · 2026-09-02T01:38:04Z

+ */
+final class CaseCatalogTest {
+  private val expectedCaseCount = 1177
+  private val expectedCatalogSha256 =


does this regenerate automatically?

Keep PR 682 focused on reusable DDL and DML coverage while moving orthogonal capabilities to extension branches. - retain 642 Parquet and ORC foundation cases - extract reusable changelog and concurrency support - preserve Plan and Scenarios consumer compatibility - add extension-stable catalog and support contract tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker and others added 5 commits August 13, 2026 12:47

docs(delta-harness): move guides to stack

4064251

Remove the harness guide and testing matrix from the implementation PR so the documentation can be reviewed in a separate stacked change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker changed the title ~~delta-harness: OpenHouse Iceberg behavioral test harness + guide~~ delta-harness: add OpenHouse behavioral test matrix Aug 25, 2026

mkuchenbecker commented Aug 26, 2026

View reviewed changes

Comment thread integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala Outdated

mkuchenbecker and others added 2 commits August 26, 2026 12:57

This was referenced Aug 26, 2026

test(delta-harness): add merge-on-read matrix mkuchenbecker/openhouse#38

Closed

test(delta-harness): add RTAS matrix mkuchenbecker/openhouse#39

Closed

test(delta-harness): add branch and WAP matrix mkuchenbecker/openhouse#40

Closed

mkuchenbecker commented Aug 26, 2026

View reviewed changes

mkuchenbecker and others added 3 commits August 27, 2026 18:02

test(delta-harness): cover cleanup failure

808fcc9

Pin the remaining ownership outcome: when the test body succeeds and cleanup fails, the cleanup failure must surface to the runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker changed the title ~~delta-harness: add OpenHouse behavioral test matrix~~ test(delta-harness): add standard matrix Sep 1, 2026

mkuchenbecker commented Sep 1, 2026

View reviewed changes

mkuchenbecker commented Sep 2, 2026

View reviewed changes

		layout.label.endsWith("/parquet") \|\|
		layout.label.endsWith("/orc"))

		val typesLayouts: List[Layout] =
		List("parquet", "orc", "avro").map(format => Layout(s"types-unpartitioned/$format", table =>

Uh oh!

Conversation

mkuchenbecker commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

Scope

Validation

Uh oh!

mkuchenbecker Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkuchenbecker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mkuchenbecker commented Aug 13, 2026 •

edited

Loading

mkuchenbecker Aug 26, 2026 •

edited

Loading