Skip to content

v0.5.560

@mkuchenbecker mkuchenbecker tagged this 13 Nov 21:30
## Summary

This PR adds an optional `WITH TIMEZONE` clause to Spark 3.5 retention
SQL. Statements without the clause continue to use UTC with unchanged
behavior. The backend policy and retention behavior are provided by [PR
#732](https://github.com/linkedin/openhouse/pull/732).

https://github.com/linkedin/openhouse/pull/735 was mistakenly merged
into a branch rather than main.

## Selecting a timezone

This policy keeps the previous 30 whole Pacific calendar days and the
current day so far:

```sql
ALTER TABLE db.events SET POLICY (
  RETENTION=30d WITH TIMEZONE 'America/Los_Angeles'
  ON COLUMN datepartition WHERE PATTERN='yyyy-MM-dd'
);
```

For a run at `2024-02-01 01:00 UTC`, Pacific time is still January 31.
The policy keeps dates `2024-01-01` through `2024-01-31`; December 31
and earlier are eligible for deletion.

## Using UTC

This statement uses UTC and removes any previously selected timezone:

```sql
ALTER TABLE db.events SET POLICY (
  RETENTION=30d
  ON COLUMN datepartition WHERE PATTERN='yyyy-MM-dd'
);
```

At the same run time, the UTC policy keeps January 2 through February 1.
January 1 is eligible for deletion because UTC has already reached
February 1.

## Validation

`WITH TIMEZONE` requires `ON COLUMN` naming a string column. Existing
default date patterns apply when `WHERE PATTERN` is omitted: a daily
policy uses `yyyy-MM-dd`.

Invalid timezones, non-string columns, and patterns that already contain
a timezone are rejected before the policy changes. Native timestamp
policies without the clause remain unchanged.

## Testing Done

- [x] The 11 timezone statement-test invocations pass on current `main`.
- [x] The 4 existing retention statement tests pass unchanged.
- [x] Spotless checks pass for the Spark 3.5 runtime and
integration-test modules.

Co-authored-by: mkuchenbecker <mkuchenbecker@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assets 2
Loading