Skip to content

v0.5.470

@mkuchenbecker mkuchenbecker tagged this 03 Aug 18:01
## Summary

`CREATE OR REPLACE ... AS SELECT` (RTAS) silently dropped the table's
policies. The replace path rebuilt the `policies` table property purely
from the incoming request, so a replace that omitted policies wiped the
existing retention, sharing, PII column tags, replication, and history,
even though ordinary user table properties survived.

## Fix

Policies are table metadata that a replace must not silently drop.
Before the replace properties are built, the existing table's policies
are merged with the request's policies. The merge is based on the
existing policies, so any plane that the request does not explicitly
provide is carried forward from the existing table, and each plane that
the request does provide overrides the existing value.

## Merge behavior

The table below describes how each policy plane behaves during a
replace.

| Plane | When the request provides it | When the request omits it |
|---|---|---|
| `retention` | The request value is applied. | The existing value is
carried forward. |
| `replication` | The request value is applied. | The existing value is
carried forward. |
| `history` | The request value is applied. | The existing value is
carried forward. |
| `lockState` | The request value is applied. | The existing value is
carried forward. |
| `columnTags` | The request map replaces the existing map in full. This
is an overwrite, not a per-key merge. | The existing map is carried
forward. An omitted field and an empty map are treated the same way. |
| `sharingEnabled` | Not applicable, because this is a primitive boolean
and a provided value cannot be distinguished from an omitted one. | The
existing value is carried forward. |

Two consequences follow from this behavior. First, because column tags
use overwrite semantics and an empty map is treated the same as an
omitted field, a replace cannot clear all column tags. Clearing tags is
done with `ALTER TABLE ... MODIFY COLUMN ... UNSET TAG`. Second, because
`sharingEnabled` is a primitive boolean with no unset state, its value
is always preserved across a replace. Sharing is changed with `ALTER
TABLE ... SET POLICY (SHARING=...)`.

Spark RTAS has no policy clause, so it always sends a request with no
policies, and the entire existing policies object is carried forward
unchanged. A partial policy payload can only arrive from a client that
calls the REST API directly.

This behavior is consistent with the intent of RTAS, which should
preserve table properties so that a replace does not require re-granting
access to the same entity.

## Testing Done

The REST level partial payload path is exercised through
`RepositoryTest`, which is the layer that can send a partial `Policies`
object. Spark cannot reach this path because it always sends a request
with no policies.

- `testReplaceMergesExistingPolicies` replaces a table without policies
and asserts that the retention policy survives.
- `testReplaceAppliesRequestedPolicies` asserts that a retention policy
provided on the request is applied.
- `testReplaceWithPartialPoliciesPreservesSharing` sends a partial
payload containing only retention and asserts that `sharingEnabled`
stays true while the new retention is applied.
- `testReplaceWithPartialPoliciesPreservesOmittedPlanes` overrides only
retention and asserts that the omitted history plane is carried forward.
- `testReplaceWithPartialPoliciesPreservesColumnTags` sends a payload
that provides retention but omits column tags, and asserts that the
existing column tag is carried forward.
- `testReplaceOverwritesColumnTags` sends a new column tag map and
asserts that it replaces the existing map in full, dropping the previous
tag.

Black box coverage is exercised through `RtasPolicyPreservationTest`
against an embedded OpenHouse server driven by Spark SQL. It asserts
that retention, sharing, the PII column tag, and history all survive a
`REPLACE TABLE ... AS SELECT`.

The existing `SnapshotsControllerTest.testPutSnapshotsReplaceCommit`
still passes, which confirms that a replace on a table that never had
policies still yields none. `./gradlew :services:tables:test` and
`:integrations:spark:spark-3.1:openhouse-spark-itest:catalogTest` pass
on JDK 17, and Spotless is clean on the module.

---------

Co-authored-by: mkuchenbecker <mkuchenbecker@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assets 2
Loading