Version 0.2 marks an important milestone in the development cycle of atmodata. With this release, atmodata can successfully be used as Minimum Viable Product (MVP).
✨ Features
- A working end-to-end pipeline for weather forecasting on WeatherBench
- A modular system that splits pipelines into datasets (e.g. WeatherBench) and tasks (e.g. Forecasting) allowing easy expansion in the future
- An extensive datapipe builder that among others allows data sharing between worker processes, and convenient creation of dataloaders.
- Fast transfer of xarray.Dataset's via shared memory to worker processes
- Full support for all variables in WeatherBench, including efficient loading and unstacking of levels.
- 21 new IterDataPipes for xarray datastructures, torch tensors, and general purpose functions, that allow easy creation of new datasets and tasks.
🌱 Planned
For the upcoming releases, the following features are planned:
- Full support for ERA5 at native resolution
- Horovod sharding support via a custom ReadingService
- Data normalization, including calculation of statistics and daily and hourly anomalies
- Documentation
- A minimum viable example demonstrating end-to-end training
- (Potential) performance optimization for collate()
- Profiling code specifically designed for datapipes.
- AtmoDistTask, a self-supervised training objective
- Refactoring
XrPrefetcherto make it more general applicable