v0.2

Version 0.2 marks an important milestone in the development cycle of atmodata. With this release, atmodata can successfully be used as Minimum Viable Product (MVP).

✨ Features

• A working end-to-end pipeline for weather forecasting on WeatherBench
• A modular system that splits pipelines into datasets (e.g. WeatherBench) and tasks (e.g. Forecasting) allowing easy expansion in the future
• An extensive datapipe builder that among others allows data sharing between worker processes, and convenient creation of dataloaders.
• Fast transfer of xarray.Dataset's via shared memory to worker processes
• Full support for all variables in WeatherBench, including efficient loading and unstacking of levels.
• 21 new IterDataPipes for xarray datastructures, torch tensors, and general purpose functions, that allow easy creation of new datasets and tasks.

🌱 Planned

For the upcoming releases, the following features are planned:

• Full support for ERA5 at native resolution
• Horovod sharding support via a custom ReadingService
• Data normalization, including calculation of statistics and daily and hourly anomalies
• Documentation
• A minimum viable example demonstrating end-to-end training
• (Potential) performance optimization for collate()
• Profiling code specifically designed for datapipes.
• AtmoDistTask, a self-supervised training objective
• Refactoring XrPrefetcher to make it more general applicable

v0.1

This release contains the very first basic set of primitive data pipes required to construct a (somewhat) complex data pipe for forecasting while still being sufficiently fast.

v0.0.2

Initial (pre-) release to claim the PyPI package name.