Introduce branching and support for spark.wap.branch

[cbb330]

Summary

Implement Iceberg table branching functionality in OpenHouse internal catalog with comprehensive testing infrastructure.

Strategically, we trust that only a valid client performs the operation. This is safe because in fact O.S. model has the client write metadata.json and send the pointer to the catalog server. So, GlueCatalog (e.g. gluetableoperations.java), DynamoDbCatalog, jdbctableoperations.java do client-only metadata operations (no server-side validation exists). The client behavior is also covered extensively by Iceberg compatibility tests applied to openhouse.

When we trust the client, we simply serialize the client request and later deserialize it. Since TableMetadata fields lack public access and only some are exposed via the builder, we start from the base metadata, take the serialized client request, and apply it to base by comparing with request and: truncating snapshot IDs from base, adding new IDs from the request, and moving the current pointer according to the request.

how to test

oh suite tests - this branch

cd /Users/chbush/code/li/project_openhouse/openhouse && \
export JAVA_HOME=$(/usr/libexec/java_home -v 17) && \
./gradlew clean \
    :integrations:spark:spark-3.5:openhouse-spark-3.5-itest:catalogTest \
    --tests "com.linkedin.openhouse.spark.catalogtest.BranchTestSpark3_5" \
    :iceberg:openhouse:internalcatalog:test \
    --tests "com.linkedin.openhouse.internal.catalog.SnapshotDiffApplierTest" \
    --tests "com.linkedin.openhouse.internal.catalog.OpenHouseInternalTableOperationsTest" \
    :integrations:spark:spark-3.1:openhouse-spark-itest:catalogTest \
    --tests "com.linkedin.openhouse.spark.catalogtest.WapIdTest" \
    :apps:openhouse-spark-apps-1.5_2.12:test \
    --tests "com.linkedin.openhouse.catalog.e2e.WapIdJavaTest"

compatibility tests - includes this branch + new shim files, and also requires this iceberg
https://github.com/cbb330/openhouse/tree/working-wap-compatibility
https://github.com/cbb330/iceberg/tree/test-harness

# 1. Clean up local maven repo to ensure fresh publish
rm -rf /Users/chbush/code/li/project_openhouse/openhouse/build && \
rm -rf /Users/chbush/.m2/repository/com/linkedin/openhouse/tables-test-fixtures* && \

# 2. Build and publish the fixtures jar (containing the new test list)
cd /Users/chbush/code/li/project_openhouse/openhouse && \
export JAVA_HOME=$(/usr/libexec/java_home -v 17) && \
./gradlew :tables-test-fixtures:tables-test-fixtures-iceberg-1.5_2.12:clean \
          :tables-test-fixtures:tables-test-fixtures-iceberg-1.5_2.12:shadowJar \
          :tables-test-fixtures:tables-test-fixtures-iceberg-1.5_2.12:publishToMavenLocal && \

# 3. Run the compatibility tests
cd /Users/chbush/code/li/project_openhouse/iceberg && \
export JAVA_HOME=$(/usr/libexec/java_home -v 17) && \
./gradlew :iceberg-core:cleanOpenhouseCompatibilityTest \
          :iceberg-spark:iceberg-spark-3.5_2.12:cleanOpenhouseCompatibilityTest \
          :iceberg-spark:iceberg-spark-extensions-3.5_2.12:cleanOpenhouseCompatibilityTest \
          :iceberg-core:openhouseCompatibilityTest \
          :iceberg-spark:iceberg-spark-3.5_2.12:openhouseCompatibilityTest \
          :iceberg-spark:iceberg-spark-extensions-3.5_2.12:openhouseCompatibilityTest \
          -PopenhouseCompatibilityCoordinate=com.linkedin.openhouse:tables-test-fixtures-iceberg-1.5_2.12:0.0.19-SNAPSHOT:uber

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

[teamurko]

Thanks @cbb330 for the PR. Would be useful to run the changes against bug bash procedures as well? Maybe those could be added as tests.

[teamurko]

Left minor comments in tests. The core logic looks good.