Skip to content

[default values] Spark catalog itest for ReadBridge initial-default overlay - #681

Open
cbb330 wants to merge 1 commit into
chbush/read-bridge-applyfrom
chbush/read-bridge-spark-itest
Open

[default values] Spark catalog itest for ReadBridge initial-default overlay#681
cbb330 wants to merge 1 commit into
chbush/read-bridge-applyfrom
chbush/read-bridge-spark-itest

Conversation

@cbb330

@cbb330 cbb330 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #679. Spark catalog itest that runs create → insert → ADD COLUMN through the OpenHouse catalog and asserts ReadBridge stamped initial-default on Spark's Iceberg table.

OSS has no deployment ColumnDefaultsSource; the local-server fixture supplies one for d1_column_default so the client overlay is exercised. SQL fill/filter cases are present but @Disabled until OpenHouse bumps Iceberg past 1.2.0.20 (linkedin/iceberg#267–269): that runtime's Spark parquet/ORC readers do not fill initial-default.

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

Adds ColumnDefaultSparkITest and a local-server ColumnDefaultsSource stub (SparkItestColumnDefaults) imported by SpringH2TestApplication.

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

./gradlew :integrations:spark:spark-3.1:openhouse-spark-itest:catalogTest --tests com.linkedin.openhouse.spark.catalogtest.ColumnDefaultSparkITest (JDK 11). Overlay test passed; SQL fill test remains disabled.

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

Stack

  1. #645 — substrate (seam + encode + decode hook)
  2. #668 — decode off Iceberg’s retry path
  3. #674 — policy / ramp in OpenHouse
  4. #678 — sanitize-on-write
  5. #679ReadBridge.apply
  6. This PR — Spark catalog itest (overlay)

Depends on #679. Enable columnDefaultBackfillViaReadBridge after the Iceberg bump that includes reader fill.

@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch 2 times, most recently from 4c6f354 to a64ea0f Compare August 13, 2026 05:52
@cbb330
cbb330 force-pushed the chbush/read-bridge-spark-itest branch from 36e818e to d799227 Compare August 13, 2026 05:59
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from a64ea0f to 4be6d59 Compare August 13, 2026 06:10
@cbb330
cbb330 force-pushed the chbush/read-bridge-spark-itest branch from d799227 to 610d668 Compare August 13, 2026 06:48
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from 4be6d59 to 08551a4 Compare August 13, 2026 22:27
@cbb330
cbb330 force-pushed the chbush/read-bridge-spark-itest branch from 610d668 to b1a506a Compare August 13, 2026 22:28
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from 08551a4 to 145297f Compare August 13, 2026 22:36
@cbb330
cbb330 force-pushed the chbush/read-bridge-spark-itest branch from b1a506a to 8c170e5 Compare August 13, 2026 22:36
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from 145297f to 17b3d21 Compare August 13, 2026 23:56
@cbb330
cbb330 force-pushed the chbush/read-bridge-spark-itest branch from 8c170e5 to d6dbba3 Compare August 13, 2026 23:56
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from 17b3d21 to d7edc72 Compare August 14, 2026 04:43
@cbb330
cbb330 force-pushed the chbush/read-bridge-spark-itest branch from d6dbba3 to a301754 Compare August 14, 2026 04:44
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from d7edc72 to 950f3f4 Compare August 14, 2026 20:41
@cbb330
cbb330 force-pushed the chbush/read-bridge-spark-itest branch from a301754 to 2228148 Compare August 14, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant