Temporal resampling: period type conversion for GeoZarr datasets (hourly→daily, daily→weekly, daily→monthly)

[turban]

Background

Several derived dataset types require period type conversion before they can be computed:

• ERA5-Land is ingested as hourly data. Daily mean/min/max temperature and daily total precipitation are far more useful for DHIS2 health applications and are required before climate normals can be computed (see Climate normals and anomalies: computation, storage, and automatic cascade #56).
• CHIRPS is daily. Monthly and weekly aggregates are needed for monthly normals, seasonal analysis, and import into DHIS2 monthly data elements.
• Any temporal aggregation pipeline (normals, anomalies, exposure indices) starts with a resampling step.

This issue defines resampling as a first-class process: a job that takes one GeoZarr as input, produces another GeoZarr at a coarser period type, and updates automatically when the upstream dataset is synced.

This is a prerequisite for #56 (climate normals and anomalies), which depends on daily ERA5 and potentially monthly CHIRPS datasets.

---

Design decisions

Output is a separate dataset, not an in-place transformation. A resampled dataset has a different period type than its source, which means a different dataset ID, different chunk shape, and different sync behaviour. Modifying the source store in place is not possible without violating the one-period-type-per-dataset constraint.

Only complete periods are written. If the source data ends mid-week, the weekly output truncates to the last complete week. This means a resampled dataset may lag the source by up to one output period. This is explicit and documented in the dataset metadata, not a silent omission.

Aggregation method is declared per dataset template. Each climate variable has a correct aggregation method: temperature uses mean (or min/max), accumulated precipitation uses sum. Declaring this in the template rather than at request time prevents incorrect aggregations from being applied silently.

---

Dataset template design

A new resample block in the YAML template declares the source and aggregation:

- id: era5land_temperature_daily
  name: ERA5-Land 2m temperature (daily mean)
  variable: t2m
  period_type: daily
  sync_kind: temporal
  sync_execution: append
  resample:
    source_dataset_id: era5land_temperature_hourly
    method: mean

- id: era5land_precipitation_daily
  name: ERA5-Land total precipitation (daily sum)
  variable: tp
  period_type: daily
  sync_kind: temporal
  sync_execution: append
  resample:
    source_dataset_id: era5land_precipitation_hourly
    method: sum

- id: chirps3_precipitation_weekly
  name: CHIRPS3 total precipitation (weekly sum)
  variable: precip
  period_type: weekly
  sync_kind: temporal
  sync_execution: append
  resample:
    source_dataset_id: chirps3_precipitation_daily
    method: sum
    week_start: monday            # ISO default; configurable

- id: chirps3_precipitation_monthly
  name: CHIRPS3 total precipitation (monthly sum)
  variable: precip
  period_type: monthly
  sync_kind: temporal
  sync_execution: append
  resample:
    source_dataset_id: chirps3_precipitation_daily
    method: sum

Supported aggregation methods

Method	Use case
mean	Temperature, humidity, wind speed
sum	Precipitation, radiation (accumulated variables)
min	Daily minimum temperature
max	Daily maximum temperature
first	Snapshot or categorical variables
last	Snapshot or categorical variables

---

Computation

New src/climate_api/processing/resample.py:

def resample_dataset(
    source_zarr_path: str,
    *,
    target_period_type: str,
    method: str,
    start: str,
    end: str,
) -> xr.Dataset:
    ds = xr.open_zarr(source_zarr_path).sel(time=slice(start, end))
    freq = _period_type_to_pandas_freq(target_period_type)  # "1D", "1W-MON", "1ME", "1YE"
    resampler = ds.resample(time=freq)
    result = getattr(resampler, method)()
    # Drop incomplete final period if source data ends mid-period
    return _drop_incomplete_trailing_period(result, source_end=end)

_drop_incomplete_trailing_period compares the last output time step against the expected end of that period. If the source data does not cover the full period, the last output step is dropped.

---

New endpoint: POST /processes/resample

POST /processes/resample
{
  "source_dataset_id": "era5land_temperature_hourly_sle",
  "target_period_type": "daily",
  "method": "mean",
  "start": "2024-01-01",
  "end": "2024-12-31"
}
→ 202 Accepted + { "job_id": "..." }

The endpoint writes the output GeoZarr and registers it as a published artifact under the resampled dataset ID. On subsequent calls for overlapping ranges it appends only the missing periods.

---

Automatic cascade on sync

When run_sync completes with status="completed" for a source dataset, the sync engine checks the dataset registry for any resampled datasets that declare that dataset as their source_dataset_id. If found, it triggers a resample job for the new delta range.

# In sync_engine.run_sync, after artifact is stored:
_trigger_downstream_resample_jobs(
    source_dataset_id=managed_dataset_id,
    delta_start=sync_detail.delta_start,
    delta_end=sync_detail.delta_end,
)

Partial period guard: the cascade passes delta_end to the resample job, which drops any trailing incomplete period. The resampled dataset's coverage end will reflect only fully covered periods, and this is recorded in the artifact metadata.

Example cascade: syncing era5land_temperature_hourly_sle through 2024-03-31T23 triggers a resample job for era5land_temperature_daily_sle covering 2024-01-01–2024-03-31. If era5land_temperature_daily_normals_sle exists (from #56), a normals computation cascade fires next.

---

Dependency order

1. resample block in dataset template YAML schema + validation
2. resample_dataset computation service
3. POST /processes/resample endpoint
4. Post-sync cascade hook in sync_engine

Steps 1–3 are sequential. Step 4 can be built in parallel with step 3 once the service exists.

---

Relationship to #56

The ERA5 daily aggregation step described in #56 (Phase 1.4) is the first use case of this process. Once this issue is implemented, #56 Phase 1.4 becomes a dataset template declaration rather than a bespoke computation step.

---

Deferred

• Downsampling (daily → hourly interpolation): not a climate use case and out of scope.
• Custom period alignment (e.g. dekadal — 10-day periods used in agricultural meteorology): the period_type enum can be extended; the xarray resample frequency mapping handles the rest.
• OGC API Processes compliance: the /processes/resample endpoint is a REST job for now, consistent with /processes/normals and /processes/anomaly from Climate normals and anomalies: computation, storage, and automatic cascade #56.

[turban]

Timezone-aware daily aggregation

ERA5-Land is stored in UTC. When aggregating hourly → daily, the day boundary must align with the local calendar day of the configured country/region, not UTC midnight. For Uganda (UTC+3), a local day runs from 21:00 UTC the previous day to 20:59 UTC — a UTC-day aggregate would span two different local days.

Where timezone lives

Timezone belongs in the extent configuration, not in each dataset template. It is a property of the deployment region, not of the variable being aggregated.

Add an optional timezone field to data/extents.yaml:

extents:
  - id: sle
    name: Sierra Leone
    bbox: [-13.5, 6.9, -10.1, 10.0]
    country_code: SLE
    timezone: Africa/Freetown   # IANA timezone identifier

timezone is optional. When absent, resampling uses UTC (appropriate for datasets where UTC-day alignment is correct or irrelevant, e.g. monthly or annual aggregates, and for deployments that explicitly want UTC days).

The ExtentRecord schema and extent loader gain a timezone: str | None field.

How resampling uses it

def resample_dataset(
    source_zarr_path: str,
    *,
    target_period_type: str,
    method: str,
    start: str,
    end: str,
    timezone: str | None = None,   # passed from extent config
) -> xr.Dataset:
    ds = xr.open_zarr(source_zarr_path).sel(time=slice(start, end))

    if timezone and target_period_type in ("daily", "weekly"):
        # Convert UTC timestamps to local time before resampling so that
        # day/week boundaries align with local calendar days.
        local_times = (
            pd.DatetimeIndex(ds.time.values)
            .tz_localize("UTC")
            .tz_convert(timezone)
        )
        ds = ds.assign_coords(time=local_times)
        result = ds.resample(time=_period_freq(target_period_type)).aggregate(method)
        # Store output as timezone-naive UTC midnight (date-only semantics)
        result = result.assign_coords(
            time=result.time.dt.tz_convert("UTC").dt.tz_localize(None)
        )
    else:
        result = ds.resample(time=_period_freq(target_period_type)).aggregate(method)

    return _drop_incomplete_trailing_period(result, source_end=end)

Output timestamp convention

After aggregation, each time coordinate represents the start of the local calendar day expressed as UTC midnight. For Sierra Leone (UTC+0 / WAT), this is a no-op. For Uganda (UTC+3), the local day 2024-03-15 is stored as 2024-03-15T00:00:00Z. This is consistent with how DHIS2 stores daily period data and is unambiguous when read back.

Monthly and annual aggregates are not affected: month/year boundaries coincide across all UTC±12 offsets in practice for health-relevant data, and xarray's ME/YE frequencies already label by first-of-period.

Extent loader change

resample_dataset receives timezone from the extent loaded at job dispatch time. The cascade hook in sync_engine reads the current instance extent to pass it through. No per-dataset template change is needed.

[turban]

Non-Gregorian calendar support (Ethiopian and Nepali)

DHIS2 supports Ethiopian (Ge'ez) and Nepali (Bikram Sambat) calendars natively. Monthly climate aggregates fed into DHIS2 must follow the same month boundaries — otherwise period codes will not align with the data elements they target.

Which aggregations are affected

Only daily → monthly (and daily → multi-month) aggregation. Weekly and daily aggregates are calendar-neutral for these deployments. Hourly → daily follows from the timezone comment above and is also unaffected by the calendar system.

Ethiopian and Nepali months do not align with Gregorian months:

Calendar	Month 1 start (approx.)	Months	Notes
Ethiopian	11 September	13 (12 × 30 days + Pagume 5–6 days)	Solar calendar, fixed offset from Gregorian
Nepali BS	Mid-April	12 (29–32 days each)	Lunisolar, lengths vary year to year

Extent configuration

Add a calendar field alongside timezone in data/extents.yaml:

extents:
  - id: eth
    name: Ethiopia
    bbox: [33.0, 3.4, 47.9, 14.9]
    country_code: ETH
    timezone: Africa/Addis_Ababa
    calendar: ethiopian

  - id: npl
    name: Nepal
    bbox: [80.1, 26.3, 88.2, 30.4]
    country_code: NPL
    timezone: Asia/Kathmandu
    calendar: nepali

calendar is optional; defaults to gregorian. Only affects monthly (and longer) aggregation.

Python libraries

• Ethiopian: convertdate — pure Python, no native deps. convertdate.ethiopian.from_gregorian(year, month, day) returns (eth_year, eth_month, eth_day). Already a common scientific Python dependency.
• Nepali: nepali-datetime — BS lookup tables. nepali_datetime.date.from_datetime_date(d) returns a BS date with .year and .month.

Implementation: groupby instead of resample

xr.resample() is Gregorian-only. For non-Gregorian monthly aggregation, replace it with a calendar-aware groupby:

def _calendar_month_labels(times: pd.DatetimeIndex, calendar: str) -> list[str]:
    """Map each UTC timestamp to its non-Gregorian YYYY-MM label."""
    if calendar == "ethiopian":
        from convertdate import ethiopian
        return [
            "{:04d}-{:02d}".format(*ethiopian.from_gregorian(t.year, t.month, t.day)[:2])
            for t in times
        ]
    if calendar == "nepali":
        import nepali_datetime
        return [
            "{:04d}-{:02d}".format(
                nepali_datetime.date.from_datetime_date(t.date()).year,
                nepali_datetime.date.from_datetime_date(t.date()).month,
            )
            for t in times
        ]
    raise ValueError(f"Unknown calendar: {calendar}")


def _aggregate_by_calendar_month(ds: xr.Dataset, method: str, calendar: str) -> xr.Dataset:
    times = pd.DatetimeIndex(ds.time.values)
    labels = _calendar_month_labels(times, calendar)
    label_da = xr.DataArray(labels, coords={"time": ds.time}, dims="time")
    grouped = ds.groupby(label_da)
    result = getattr(grouped, method)()
    # Rename the groupby dimension back to "time"
    return result.rename({"group": "time"})

Output time coordinate

The time coordinate in the output Zarr stores the calendar month label as a string ("2016-01", "2081-04") rather than a datetime. This is the only unambiguous representation: there is no Gregorian datetime that is "Ethiopian month 1" without a lookup.

The Zarr time variable attributes must record the calendar:

result["time"].attrs["calendar"] = calendar          # "ethiopian" or "nepali"
result["time"].attrs["axis"] = "T"
result["time"].attrs["long_name"] = "time"

Any downstream consumer (DHIS2 integration in Step 2, climate normals in #56) reads the calendar attribute and handles the label format accordingly.

The 13th Ethiopian month (Pagume)

Pagume has 5 days (6 in Ethiopian leap years). It is a valid output period and must not be silently dropped. The _drop_incomplete_trailing_period guard must be Pagume-aware: a Pagume period with fewer than 5 source days present is incomplete; with 5 or 6 days present it is complete.

Impact on shared/time.py

datetime_to_period_string and normalize_period_string are currently Gregorian-only. They remain unchanged — they operate on dataset-native period strings for the ingestion/sync layer, which is always Gregorian (ERA5, CHIRPS). Calendar conversion is a resampling concern only and lives in processing/resample.py.