v2.1.0: Full DWD history, rate-limit-safe OpenWeather

A weather-scraper release that makes backfills more reliable. DWD backfills now really return the full station history, instead of quietly stopping after about 500 days. OpenWeather backfills now handle rate limits: they retry on HTTP 429 instead of leaving permanent gaps.

Bug fixes

• DWD history backfill: before, the URL of the historical archive was built from the station's end date in the recent station list. For active stations that is yesterday, but the archive ends at the yearly cutoff (e.g. 20251231), so the request always returned 404. The provider then fell back to recent data only (~500 days) without any error. The filename is now looked up from the archive's directory listing.
• DWD station names with umlauts: the station list is ISO-8859-1 but was read as UTF-8, so names like Düsseldorf showed up as D�sseldorf in logs. As a side effect, Baden-Württemberg and Thüringen were not recognised, which affected 316 of 1386 stations. The list is now decoded when it is downloaded.

Changes

• OpenWeather retries on HTTP 429: a 429 is retried up to 3 times with exponential backoff (30s, 60s, 120s). This covers OpenWeather's per-minute window. Before, the day was logged and skipped. The retry applies to backfills and to the daily scrape. Waiting stops when the context is cancelled.
• OpenWeather backfill requires --min-delay ≥ 1: --min-delay 0 turned off the sleep between requests and could exceed the limit of 60 calls per minute. backfill --provider openweather now rejects values below 1 before connecting to the database. The default is already 1.

Tip

If you backfilled with DWD more than ~500 days back on v2.0.x, the older range was not stored. Run the backfill again to fill the gap.

No schema changes. The oil scraper is unchanged. The only behaviour change is that OpenWeather backfills reject --min-delay 0.

Full commit list

• d8a85b4 Decode DWD station list as ISO-8859-1
• 3d4d568 Decode station list at download and use real UTF-8 fixture
• ef3912b Look up DWD historical ZIP filename from directory listing
• 362a2e3 Require minimum backfill delay of 1s for OpenWeather
• d43f309 Retry OpenWeather requests on HTTP 429 with backoff

v2.0.1: No more re-scrapes on restart

A small bug-fix release for the oil scraper. Before, every restart fetched Hoyer prices again, even when today's prices were already stored. The "already scraped today?" check now takes the zip code into account, and it works for every provider.

Bug fixes

• Startup check now works for local providers: before, HasScrapedToday always returned false for zip-code-based providers like Hoyer, so they were re-fetched on every restart. It now checks the provider's zip code.
• Check no longer limited to one product type: before, it only looked for the standard product type, which only HeizOel24 uses. It now matches any price for that provider, date and zip code.
• Correct with shared databases: several instances with different --zip-code values can share one database, and each one's check stays correct.

No breaking changes, config changes or schema changes.

Full commit list

• 32fb232 Check scraped-today status per zip code for local providers

v2.0.0: Oil meets weather, now called heizsaison

The project now covers more than oil prices. v2.0.0 adds a second binary that collects daily weather data from five providers, so you can relate heating oil prices to the actual heating season. To match, the whole project is renamed to heizsaison: module, binaries, images, metrics and database. Many internals were also consolidated.

⚠️ Breaking changes

• Go module: github.com/andygrunwald/oil-price-scraper → github.com/andygrunwald/heizsaison. go install users must switch to the new path.
• Binaries: oilscraper / weatherscraper → heizsaison-oil / heizsaison-weather
• Docker images: ghcr.io/andygrunwald/oil-price-scraper → ghcr.io/andygrunwald/heizsaison-oil (weather: heizsaison-weather)
• Prometheus metrics: oilscraper_* → heizsaison_oil_*, weatherscraper_* → heizsaison_weather_*. Old series are not aliased, so dashboards and alerts must be updated.
• Database and role: oil / oilscraper → heizsaison. Migrate before upgrading:

  ALTER DATABASE oil RENAME TO heizsaison;
  ALTER ROLE oilscraper RENAME TO heizsaison;

  Table names oil_prices and weather_observations are unchanged.
• Config behaviour: the SCRAPE_HOUR and PROVIDERS env vars now take effect (before, they were silently ignored). Check your env for values you didn't intend.

Features

• New weather binary with the same commands (run, scrape, backfill, version), its own Docker image and a Compose service on port 8081
• 5 weather providers:
  • Open-Meteo: no API key, history back to 1940
  • Bright Sky: DWD data, aggregated from hourly to daily
  • Visual Crossing: API key required
  • OpenWeather One Call 3.0: API key required
  • DWD CDC-OpenData: official station data, automatic nearest-station lookup, 50+ years of history
• New weather_observations table
• --providers accepts repeated or comma-separated values

Bug fixes

• run/scrape/backfill flags are now bound to the config, so SCRAPE_HOUR, PROVIDERS and the backfill config are no longer dead
• CLI help no longer says "MySQL"

Internal and docs

• Merged duplicate packages (weatherscraper, weatherapi, weatherhttp, weatherconfig) into shared ones. Added the internal/numeric helpers and the first scraper tests, including a race test.
• README rewritten. New docs/ for configuration, HTTP endpoints, development, providers, debug logging and example SQL queries.

Full commit list

• 3d5ce15 Add DWD CDC-OpenData weather provider
• db0e687 Add weather data scraping support with 4 providers
• 2c268ec Bump github.com/jackc/pgx/v5 from 5.8.0 to 5.9.2
• 6f31bd4 Check error return value of deferred Close calls
• 8d871e3 Consolidate the four float64Ptr copies into internal/numeric
• d4f2548 Document how to enable debug logging
• c118df4 Extract HTTP endpoint documentation into docs/HTTP_ENDPOINTS.md
• 406687d Extract configuration reference into docs/CONFIGURATION.md
• 86d3453 Extract development guide into docs/DEVELOPMENT.md
• 35b197f Extract oil price operations into internal/database/oil.go
• 825eafb Extract oil price types into internal/models/oil.go
• ee5c4b9 Extract slice aggregation helpers into internal/numeric
• 923edc9 Give each dockers_v2 block a unique ID
• 4a3717c Ignore the weatherscraper binary
• 50b6f62 Merge internal/weatherapi into internal/api
• 31c2bec Merge internal/weatherconfig into internal/config
• 9851fc8 Merge internal/weatherhttp into internal/http
• 4c37ddc Merge internal/weatherscraper into internal/scraper
• 182d1c6 Pin golangci-lint and track it with Renovate
• 487e089 Pin the GitHub Actions runners to ubuntu-24.04
• 2af082d Pin the goreleaser CLI version
• 711310f Rename 001_initial_schema.sql to 001_oil_schema.sql
• d9f06d5 Rename Go module and binaries to heizsaison
• de2d4fa Rename Prometheus metric namespaces to heizsaison
• b5ee022 Rename migrations/ to database-schema/
• 41fd428 Rename the Postgres database and role to heizsaison
• 263bfb5 Rewrite README.md as a concise entry point
• b26c990 Run CI checks through the make targets
• 8462903 Track staticcheck as a Go tool dependency
• 2141606 Update CLI help and package docs for the heizsaison rename
• 149551f Update README and documentation for the heizsaison rename
• 041f956 Update actions/checkout action to v7
• 073cac9 Update actions/setup-go action to v7
• 8eae3bf Update alpine Docker tag to v3.23
• 64e6c3e Update alpine Docker tag to v3.24
• 6a1cb6c Update alpine Docker tag to v3.24
• 2aba9ba Update codecov/codecov-action action to v7
• aa48d8d Update dependency go to 1.27
• ab1a82f Update dependency golangci/golangci-lint to v2.14.0
• 736e389 Update dependency goreleaser/goreleaser to v2.18.3
• 166cbf8 Update dependency ubuntu to v26
• dc3972a Update module github.com/jackc/pgx/v5 to v5.10.0
• 155b1f3 Update module github.com/jackc/pgx/v5 to v5.11.0
• f210008 Update module github.com/prometheus/client_golang to v1.24.1
• d020237 Update module github.com/prometheus/client_golang to v1.25.0
• 5bde257 Update module github.com/rs/zerolog to v1.35.1
• 85bf2d0 Update postgres Docker tag to v18.6
• 6d368b9 Wire the run, scrape and backfill flags to the config
• 9f52417 build: Rename build targets to build-oilscraper and build-weatherscraper
• f2c7f31 chore(deps): update dependency go to 1.26
• 7a1ff86 chore(deps): update docker/login-action action to v4
• 5010844 chore(deps): update docker/setup-buildx-action action to v4
• e848b8d chore(deps): update goreleaser/goreleaser-action action to v7
• 161d12b go fmt ./...

v1.0.1: Real prices, leaner storage

A patch release. Hoyer prices now include VAT, so they match what consumers actually pay. Raw API responses are no longer stored by default.

Bug fixes

• Hoyer: store the gross price (priceGross) instead of the net basePrice, so the value per 100 L includes taxes

Changes

• Default config: StoreRawResponse is now false. Set it explicitly if you need the raw payloads.

Note

Hoyer prices stored by v1.0.0 (net) and v1.0.1 (gross) are not directly comparable in historical data.

Full commit list

• 5d0c234 Default config: Set StoreRawResponse to false
• fc0fe90 Hoyer: Use gross price (priceGross) instead of net price (basePrice)

v1.0.0: Germany's heating oil prices, scraped daily

The first public release of a Go service that collects German heating oil prices from two providers every day and stores them in PostgreSQL.

Features

• Oil price scraping from HeizOel24 (national average, with historical backfill) and Hoyer (regional prices by zip code, all product variants)
• PostgreSQL storage with duplicate prevention
• HTTP endpoints: /metrics (Prometheus: API latency, last scrape, current prices, DB operations) and /status (scheduler state, last scrape)
• Docker image, plus a Docker Compose dev stack

Full commit list

• 4a27669 Add .claude to gitignore and remove from tracking
• 84dd2cb Add Goreleaser for automated releases and Docker publishing
• 31d276f Add Makefile with build, test, and code quality targets
• aa03e20 Add renovate.json
• 738fdc0 Cleanup unused functions and wire Prometheus metrics
• e237368 Extract Cobra commands into separate files
• 4b06866 Fix GoReleaser deprecations and pin version to v2.13
• 7aeebef Fix Hoyer API JSON struct to match actual response format
• ef040ac Fix PostgreSQL 18+ volume mount path
• 00d9a9e Fix errcheck linting errors by panicking on error
• 7fe3b0b Github Actions/Lint: Fix .golangci.yml
• 0ab53be Github Actions: Add workflow_dispatch for manual trigger
• 237409d Improve HeizOel24 provider: use yesterday-to-now range and rename url variable
• 7f6f907 Initial implementation of heating oil price scraper
• 0e32b58 Make --zip-code a required flag
• afaf069 Migrate from MySQL to PostgreSQL 18.1
• 3403c46 README: Add --zip-code to command examples
• 6a48c00 README: Rename database from oilprices to oil
• 1d9117f Remove obsolete version attribute from docker-compose.yml
• f6ed4a7 Update MySQL to 8.4.7 LTS
• 42691b0 Update actions/checkout action to v6
• e63aea3 Update actions/setup-go action to v6
• 48c9300 Update codecov/codecov-action action to v5
• 098a4cc Update golangci-lint config for v2 compatibility
• 5f672ce Update golangci/golangci-lint-action action to v9
• 54e74ee Upgrade Go version to 1.25
• e087a29 Use scratch as Docker base image for minimal footprint
• 54c2e49 Wrap MySQL schema identifiers in backticks
• 52b066d go get -u ./... && go mod tidy