Skip to content

v0.2.18

Choose a tag to compare

View all tags
@kayhendriksen kayhendriksen released this 17 Apr 18:05
· 78 commits to main since this release
v0.2.18
f00f049

0.2.18 — Parallel downloads

Downloads are now fetched concurrently per dataset using a thread pool. A collection with hundreds of station CSVs (e.g. smn) is significantly faster, while staying polite to the MeteoSwiss / CSCS CDNs.

Changes

  • Per-asset downloads run concurrently in download_collection, download_metadata, download_grib2, and download_netcdf.
  • New --workers N flag on foehn download (default: 8).
  • foehn.download(...) Python API gains a workers keyword argument.
  • A single requests.Session is reused across threads per dataset, instead of being rebuilt per request.

Usage

foehn download smn                  # default 8 concurrent downloads
foehn download smn --workers 16     # bump it up if your connection allows
foehn download smn --workers 1      # serial, old behaviour
Assets 2
Loading