Skip to content

v2.1.0: Full DWD history, rate-limit-safe OpenWeather

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 11 Oct 18:31
· 3 commits to main since this release
v2.1.0
This tag was signed with the committer’s verified signature.
andygrunwald Andy Grunwald
GPG key ID: 4DF3757FE7EB0B6E
Verified
Learn about vigilant mode.
b182ee5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

A weather-scraper release that makes backfills more reliable. DWD backfills now really return the full station history, instead of quietly stopping after about 500 days. OpenWeather backfills now handle rate limits: they retry on HTTP 429 instead of leaving permanent gaps.

Bug fixes

  • DWD history backfill: before, the URL of the historical archive was built from the station's end date in the recent station list. For active stations that is yesterday, but the archive ends at the yearly cutoff (e.g. 20251231), so the request always returned 404. The provider then fell back to recent data only (~500 days) without any error. The filename is now looked up from the archive's directory listing.
  • DWD station names with umlauts: the station list is ISO-8859-1 but was read as UTF-8, so names like Düsseldorf showed up as D�sseldorf in logs. As a side effect, Baden-Württemberg and Thüringen were not recognised, which affected 316 of 1386 stations. The list is now decoded when it is downloaded.

Changes

  • OpenWeather retries on HTTP 429: a 429 is retried up to 3 times with exponential backoff (30s, 60s, 120s). This covers OpenWeather's per-minute window. Before, the day was logged and skipped. The retry applies to backfills and to the daily scrape. Waiting stops when the context is cancelled.
  • OpenWeather backfill requires --min-delay ≥ 1: --min-delay 0 turned off the sleep between requests and could exceed the limit of 60 calls per minute. backfill --provider openweather now rejects values below 1 before connecting to the database. The default is already 1.

Tip

If you backfilled with DWD more than ~500 days back on v2.0.x, the older range was not stored. Run the backfill again to fill the gap.

No schema changes. The oil scraper is unchanged. The only behaviour change is that OpenWeather backfills reject --min-delay 0.

Full commit list
  • d8a85b4 Decode DWD station list as ISO-8859-1
  • 3d4d568 Decode station list at download and use real UTF-8 fixture
  • ef3912b Look up DWD historical ZIP filename from directory listing
  • 362a2e3 Require minimum backfill delay of 1s for OpenWeather
  • d43f309 Retry OpenWeather requests on HTTP 429 with backoff
Assets 7
Loading