Skip to content

change source interval in the export package#2265

Merged
iluise merged 2 commits into
developfrom
iluise/develop/hourly-quaver
Apr 27, 2026
Merged

change source interval in the export package#2265
iluise merged 2 commits into
developfrom
iluise/develop/hourly-quaver

Conversation

@iluise
Copy link
Copy Markdown
Contributor

@iluise iluise commented Apr 24, 2026

Description

implement a more solid way to retrieve the forecast steps that takes into account also the substeps.

init_time = valid_time - source_end

Where source_end is the last valid_time of the source window.

example:

uv run export --run-id b9i7napw --output-dir ./results/b9i7napw/ --format quaver --stream ERA5 --type prediction --samples 0 1 --fsteps 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32  --quaver-template-folder ./grib_templates/ --quaver-template-grid-type o96 --expver j3xx --n-processes 48 --channels 2t msl 10u 10v t_500 t_850 z_500 z_850 q_500 q_850 v_500 v_850 u_500 u_850
Dimensions:        (time: 2, step: 60, isobaricInhPa: 2, values: 40320)
Coordinates:
  * time           (time) datetime64[ns] 16B 2023-10-01T05:00:00 2023-10-01T1...
  * step           (step) timedelta64[ns] 480B 07:00:00 ... 5 days 00:00:00
    valid_time     (time, step) datetime64[ns] 960B ...
  * isobaricInhPa  (isobaricInhPa) float64 16B 850.0 500.0
    latitude       (values) float64 323kB ...
    longitude      (values) float64 323kB ...
Dimensions without coordinates: values
Data variables:
    q              (time, step, isobaricInhPa, values) float32 39MB ...
    u              (time, step, isobaricInhPa, values) float32 39MB ...
    v              (time, step, isobaricInhPa, values) float32 39MB ...
    z              (time, step, isobaricInhPa, values) float32 39MB ...
    t              (time, step, isobaricInhPa, values) float32 39MB ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2026-04-24T10:40 GRIB to CDM+CF via cfgrib-0.9.1...

steps:

> ds.step.astype("timedelta64[ns]")/3600000000000
<xarray.DataArray 'step' (step: 60)> Size: 480B
array([  7,   8,   9,  10,  11,  12,  19,  20,  21,  22,  23,  24,  31,
        32,  33,  34,  35,  36,  43,  44,  45,  46,  47,  48,  55,  56,
        57,  58,  59,  60,  67,  68,  69,  70,  71,  72,  79,  80,  81,
        82,  83,  84,  91,  92,  93,  94,  95,  96, 103, 104, 105, 106,
       107, 108, 115, 116, 117, 118, 119, 120], dtype='timedelta64[ns]')

Issue Number

Closes #2264

Is this PR a draft? Mark it as draft.

Checklist before asking for review

  • I have performed a self-review of my code
  • My changes comply with basic sanity checks:
    • I have fixed formatting issues with ./scripts/actions.sh lint
    • I have run unit tests with ./scripts/actions.sh unit-test
    • I have documented my code and I have updated the docstrings.
    • I have added unit tests, if relevant
  • I have tried my changes with data and code:
    • I have run the integration tests with ./scripts/actions.sh integration-test
    • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
    • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
  • I have informed and aligned with people impacted by my change:
    • for config changes: the MatterMost channels and/or a design doc
    • for changes of dependencies: the MatterMost software development channel

@github-actions github-actions Bot added the eval anything related to the model evaluation pipeline label Apr 24, 2026
@iluise iluise requested a review from enssow April 24, 2026 08:52
@iluise iluise changed the title change source interval change source interval in the export package Apr 24, 2026
Copy link
Copy Markdown
Contributor

@enssow enssow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with both netcdf and verif parser using id=cae70ju8; s=ERA5; uv run export --run-id $id --stream $s --output-dir ../${id}_${s} --format verif --obs ../metno_observations_v3.nc and id=cae70ju8; s=ERA5; uv run export --run-id $id --stream $s --output-dir ../${id} --format netcdf on SANTIS; both produced expected results

@iluise iluise merged commit f716c23 into develop Apr 27, 2026
5 of 6 checks passed
wael-mika pushed a commit to wael-mika/WeatherGenerator that referenced this pull request May 3, 2026
* change source interval

* fix chunking
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval anything related to the model evaluation pipeline

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

adapt export package for hourly ERA5

2 participants