Skip to content

v0.3.0: Memory efficiency and Metadata Convention

Choose a tag to compare

@heikoklein heikoklein released this 22 Sep 11:24
· 19 commits to main since this release

The 0.3.0 release enables a more memory efficient data-structure for the Data-section of pyaro, identifying stations by integer ids rather than full station -names. In addition, pyaro has now a Metadata Convention, to avoid metadata fragmentation.

For this the interface has been extended with:

  • add Data.station_ids
  • add Data.unique_by_keys()
  • removed requirement for Data.keys()
  • add new implementation of Data, called DataStationIdStructured separating stations and measurements in a memory efficient way
  • Metadata-Convention
  • bump minor version to 0.3 to indicate api-changes

The new DataStationIdStructured is a drop-in replacement for NpStructuredData. But rather than repeating all
columns like station, latitude, longitude, altitude for each data-point (hourly measurement), it uses internally two separate datatables, one for the station-information and one for the time-series data-point.

A single additional measurement for an existing station will be reduced from 310 bytes to 30 bytes. A new station will require 268 bytes + an dictionary entry (string(256bytes) to int(4 bytes)) mapping. Per one station (overhead estimated as 1kB):

NpStructuredData DataStationIdStructured
1 year of hourly data 2652 kB 257 kB
1 year of daily data 110 kB 11kB
1 year of monthly data 3.6 kB 1.4 kB

What's Changed

Full Changelog: v0.2.1...v0.3.0