Context
DJ currently supports day-level granularity for temporal partition columns. We should add support for hour granularity so cubes and pre-aggregations can refresh on an hourly cadence.
The Granularity enum already includes HOUR and downstream metadata (lookback defaults, cron defaults, pre-aggregation model) supports it. However, cube materialization SQL generation hardcodes day-level format, so setting granularity: hour on a cube's temporal partition does not produce hour-aware filter SQL.
User Outcome
After this lands, a user should be able to:
- Define a cube with an hour-grain temporal partition column in YAML:
columns:
- name: dateint_hour
type: int
partition:
type: temporal
granularity: hour
format: yyyyMMddHH
- Configure materialization at hourly cadence -- the platform reads
granularity: hour and applies the right defaults:
lookback_window: 1 HOUR
- Cron schedule:
0 * * * * (top of every hour)
- Workflow iterates per-hour over the backfill range
- Get hour-aware filter SQL generated automatically. Calling the materialization endpoint produces partition filters like:
dateint_hour = CAST(DATE_FORMAT(CAST(DJ_LOGICAL_TIMESTAMP() AS TIMESTAMP), 'yyyyMMddHH') AS INT), or a BETWEEN form with the configured lookback for catch-up runs.
Context
DJ currently supports day-level granularity for temporal partition columns. We should add support for hour granularity so cubes and pre-aggregations can refresh on an hourly cadence.
The
Granularityenum already includesHOURand downstream metadata (lookback defaults, cron defaults, pre-aggregation model) supports it. However, cube materialization SQL generation hardcodes day-level format, so settinggranularity: houron a cube's temporal partition does not produce hour-aware filter SQL.User Outcome
After this lands, a user should be able to:
granularity: hourand applies the right defaults:lookback_window: 1 HOUR0 * * * *(top of every hour)dateint_hour = CAST(DATE_FORMAT(CAST(DJ_LOGICAL_TIMESTAMP() AS TIMESTAMP), 'yyyyMMddHH') AS INT), or aBETWEENform with the configured lookback for catch-up runs.