Repository navigation
v0.3.0: Memory efficiency and Metadata Convention
The 0.3.0 release enables a more memory efficient data-structure for the Data-section of pyaro, identifying stations by integer ids rather than full station -names. In addition, pyaro has now a Metadata Convention, to avoid metadata fragmentation.
For this the interface has been extended with:
- add
Data.station_ids - add
Data.unique_by_keys() - removed requirement for
Data.keys() - add new implementation of
Data, calledDataStationIdStructuredseparating stations and measurements in a memory efficient way - Metadata-Convention
- bump minor version to 0.3 to indicate api-changes
The new DataStationIdStructured is a drop-in replacement for NpStructuredData. But rather than repeating all
columns like station, latitude, longitude, altitude for each data-point (hourly measurement), it uses internally two separate datatables, one for the station-information and one for the time-series data-point.
A single additional measurement for an existing station will be reduced from 310 bytes to 30 bytes. A new station will require 268 bytes + an dictionary entry (string(256bytes) to int(4 bytes)) mapping. Per one station (overhead estimated as 1kB):
| NpStructuredData | DataStationIdStructured | |
|---|---|---|
| 1 year of hourly data | 2652 kB | 257 kB |
| 1 year of daily data | 110 kB | 11kB |
| 1 year of monthly data | 3.6 kB | 1.4 kB |
What's Changed
- Memory efficient data-structure by @heikoklein in #93
- Metadata conventions for pyaro by @heikoklein in #92
Full Changelog: v0.2.1...v0.3.0