v0.18.0

Earth2Studio general release 0.18.0

🚨 Breaking Changes 🚨

• No notable breaks this release 🎉

📕 Changelog

🌎 Added

• Added model scorecards to the documentation produced by the new recipes/eval/scorecard campaigns (FCN3 and Aurora to start)
• Added IEM parsed ASOS/AWOS station observation data source (IEM_ASOS)
• Added IceChunkBackend IO backend for writing versioned, transactional output to an Icechunk repository, with non-blocking writes by default (blocking=True to opt back into synchronous writes)
• ZarrBackend (and IceChunkBackend) write/read now use basic slice indexing for contiguous coordinate subsets, ~7x faster than the previous fancy-indexing path on step-by-step forecast writes
• Added SamudrACE coupled atmosphere-ocean prognostic model (SamudrACE) with its initial-condition and forcing data sources (SamudrACEData, SamudrACEForcingData)
• Added CorrDiffCosmoEra5SDA, score-based data assimilation (DPS) for the CorrDiff-COSMO downscaler
• Added Zarr v3 sharding support to AsyncZarrBackend
• Added obstore support to AsyncZarrBackend via a new store parameter (store URL, obstore store, or zarr store)
• Added a working add_array plus __contains__, __getitem__, __iter__, __len__, store and coords to AsyncZarrBackend, matching ZarrBackend, and output.io_backend to the eval recipe to select between them (async_zarr is the new default)
• Added hyperspectral IR sounder variables (airs, iasi, cris) to NNJAObsSat, returned as brightness temperature (K) with per-channel wavenumbers alongside the existing microwave sensors
• Added Atlas CRPS ensemble prognostic model (AtlasCRPS), which generates ensemble members from noise conditioned transformer blocks and shares the Atlas autoencoder
• Added earth2studio.data.utils.table_to_dataframe, a shared Arrow-to-pandas conversion producing fully Arrow-backed (pd.ArrowDtype) DataFrames with optional dictionary encoding of low-cardinality string columns
• Added StormScopeMeteoSatEU European domain satellite nowcasting model

⚙️ Changed

• GFS, HRRR, and GEFS GRIB sources now fill unresolved output regions with NaN instead of zeros or uninitialized memory, making missing index records detectable
• UFSObsConv and UFSObsSat now decode diag files in parallel across a persistent spawn-based process pool (new decode_workers parameter, default "auto").
• Updated StormCast SDA example to use the GHCNHourly data source.
• Vectorized the NNJAObsSat IR sounder decode (one NumPy pass per footprint instead of per-channel operations; IASI ~7.8x, CrIS ~5.8x faster) and moved decode workers to per-batch Arrow tables instead of pickled row dicts (~6.2x less accumulation memory, near-zero cross-process serialization cost)
• NNJAObsSat output DataFrames now use Arrow-backed dtypes for every column across both microwave and IR sensors: floats as float[pyarrow]/ double[pyarrow], times as timestamp[ns][pyarrow], and the satellite, variable, and class string columns dictionary-encoded
• AsyncZarrBackend now throttles on in flight writes rather than submitted writes, and waits for whichever write completes first rather than the oldest.
• Migrated GOES data source from s3fs to obstore; hour-directory listings are now async and memoized, so same-hour timestamps share one LIST request (~30% faster)
• Migrated GOES GLM data source from s3fs to obstore; listings of complete hours are memoized per instance while the current hour is always re-listed
• Migrated Himawari AHI data source from s3fs to obstore with memoized minute-directory listings (scans older than an hour)
• Migrated Planetary Computer data sources from planetary_computer.sign() + httpx streaming to obstore AzureStore with the
  PlanetaryComputerCredentialProvider, which fetches and renews SAS tokens automatically; the planetary-computer and httpx packages are no longer required by the data extra
• PlanetaryComputerECMWFOpenDataIFS now downloads only the GRIB messages for the requested variables via byte ranges resolved from the item's GRIB index asset (~6x faster and ~25-180x less transfer than the previous whole-file download, depending on variable count)
• obstore_fetch_to_cache gained an opt-in atomic mode that publishes cache files via temp file + rename, so interrupted or concurrent downloads cannot leave a partial file as a poisoned cache entry; used by the Planetary Computer sources' fetches. Off by default, leaving the other obstore-migrated data sources' cache writes unchanged.
• Migrated GHCNDaily and GHCNHourly data sources to obstore; GHCNDaily station-scale requests now fetch per-station parquet files instead of global by_year partitions (~25x faster), and their default async_workers is raised from 16 to 32 since fetches are small, latency-bound requests where throughput scales with concurrency (roughly halves wall time again for large station lists)
• Migrated JPSS VIIRS, ATMS, and CrIS data sources from s3fs to obstore; day-directory listings of completed days are memoized per instance while in-progress days are always re-listed, and the CrIS SDR/GEO dual listings remain concurrent
• Vectorized the JPSS ATMS BUFR decode (numpy column assembly + Arrow table accumulation instead of per-row dicts), roughly halving end-to-end fetch time; decoded output is bit-identical to the previous implementation
• JPSS VIIRS HDF5 decode now runs in worker threads (serialized by an HDF5 lock) so decoding no longer blocks concurrent granule downloads
• JPSS ATMS now decodes each BUFR file as soon as its download completes (pipelined with in-flight downloads, one decode per unique file) instead of decoding after all downloads finish
• JPSS CrIS granule downloads now fetch large objects as concurrent byte ranges (1 MiB x 8 streams per file) instead of one single-stream GET, roughly halving fetch time for typical requests
• Migrated ISD, IBTrACS, CFS reforecast, and OPERA data sources from fsspec/s3fs to obstore
• CFS reforecast grib decoding now resolves all requested variables in a single pass over the file's messages instead of one pygrib.select scan per variable (~20x faster for full-lexicon requests)
• Consolidated the four identical per-source grib decode helpers (_decode_gfs_grib, _decode_hrrr_grib, _decode_gefs_grib, _decode_cfs_grib) into a shared decode_grib_message helper
• Migrated MRMS data source from s3fs to obstore, with memoized day-directory listings and threaded, header-based grid decoding
• Migrated NClimGridDaily data source from s3fs to obstore; monthly NetCDF files are now downloaded once into the cache and shared across all (day, variable) slices instead of being streamed per slice over fsspec
• Updated DLESyM and DLESyMLatLon default package to provide newer CRPS-trained checkpoints used in the AI Weather Quest competition

❌ Removed

• Deprecated the fs_factory parameter of AsyncZarrBackend in favor of store; the default local write path no longer uses fsspec

🔨 Fixed

• Fixed intermittent repeated FengWu forecast timesteps by synchronizing ONNX Runtime I/O buffers with PyTorch.
• Fixed HRRR GRIB index lookups for total precipitation at lead zero and for snow depth, snow cover, and total cloud cover at positive forecast lead times.
• Fixed the March 22, 2021 GFS archive cutoff and historical GFS_FX total precipitation lookups.
• Fixed lat_weight returning a small negative weight at the poles in float32, which could give NaN under sqrt. Weights are now clamped to be non-negative.
• Fixed SamudrACE inference being non-reproducible run-to-run: toggling torch.backends.cudnn.benchmark on and off around each coupled cycle re-triggered cuDNN's GPU-timing-based algorithm search every cycle
• Fixed Aurora.create_iterator first yield pairing a lead-sliced tensor with unsliced coords: lead_time kept [-6h, 0h] while the tensor held one step. Coords are now sliced the same way as FuXi, DLWP and FengWu, so the initial condition is yielded at lead_time=[0h]
• Fixed CFS_Reforecast_FX and CFS_Reforecast_FX_Flux pointing at the retired NCEI archive path; the reforecast archive moved to https://www.ncei.noaa.gov/oa/prod-cfs-reforecast with renamed product subdirs
• Fixed CorrDiffCosmoEra5 loading files from the wrong resolution when cache names collided. Cache names now include the resolution.
• Fixed OPERA data source returning negative precipitation values (-99.0 mm/h for tprate, -0.099 m for tp01) for pixels where the radar detected no rain. Undetect pixels for RATE and ACRR quantities are now filled with 0.0 instead of the reflectivity sentinel -99.0 dBZ.
• Fixed GHCNHourly station discovery to use the published GHCNh station list (ghcnh-station-list.csv) instead of the GHCN-Daily station list.
• Fixed to_time_array and fetch_data silently wrapping timestamps outside the datetime64[ns] range (roughly 1678-2262) to unrelated dates, which prevented workflows such as run.deterministic from using the model-year calendars of climate emulators (for example the CM4 initial conditions of SamudrACE).
• Fixed AIFS2 and AIFS2ENS assigning time-dependent forcing values to the wrong samples when processing multiple batches and initialization times.
• Fixed AsyncZarrBackend discarding exceptions raised by non-blocking writes. A write future that had already completed was never resulted, so its error was swallowed
• Fixed AsyncZarrBackend bugs covering non-blocking write safety, tensor aliasing, metadata visibility, coordinate parsing, and shard buffer allocation.
• Fixed Atlas models using incorrect total precipitation accumulation, now models correctly use tp06.

Security

• Added zizmor static security auditing of GitHub Actions workflows (pre-commit hook, make zizmor lint step, and a code scanning workflow)

📦 Dependencies

• Added obspec>=0.1 core dependency; the shared obstore helpers are typed against its vendor-neutral store protocols
• Updated GraphCast and GenCast optional dependencies to use WeatherNext.

Special thank you to our external contributors:

• Elynn Wu
• Ian