Skip to content

SMART Data Updates: Dynamic Dataset Management & Coverage-Based Forecast Filling

Choose a tag to compare

View all tags
@danishru danishru released this 20 Jun 22:38
v0.4.0
92508a8

🧠 Smarter Updates, Continued

This release continues the SMART approach introduced earlier and focuses on making updates more thoughtful and efficient.

SILAM Pollen now better understands what data is actually available and updates forecasts accordingly — avoiding unnecessary requests and keeping everything fast and predictable.

In this update, the same philosophy is extended to runtime dataset management, making the coordinator more resilient to catalog changes and improving long-term maintainability.

SILAM Pollen v0.4.0 Release Notes

🚀 What’s New

  • 🗂️ Shared runs catalog manager

    • Introduced a shared runs catalog manager used by all coordinators.
    • Centralizes parsing and caching of runs/catalog.xml.
    • Avoids duplicate catalog requests when multiple locations or datasets are used.
    • Runs catalog data is cached for 45 minutes.

  • 🧠 Coverage-aware SMART dataset management (preferred + fallback)

    • Catalog coverage is treated as the real available forecast range (start → earliest, end → latest).
    • SMART now manages datasets continuously at runtime:
      • switches to an alternative dataset when the selected one cannot satisfy the expected forecast window,
      • safely returns to the preferred dataset once it publishes a fresh run_id with sufficient coverage.
    • Keeps updates predictable while staying resilient during catalog/run transitions.

  • 🏷️ Dataset source tracking

    • Forecast data is now internally tagged with the dataset it was sourced from.
    • Enables reliable diagnostics and lays the groundwork for multi-dataset logic.

  • ✂️ Smarter incremental updates (coverage-aware + priority fill)

    • Incremental updates now avoid extra range fetches when the catalog indicates no additional data is available.
    • If the preferred dataset does not provide enough coverage to reach the configured forecast horizon, SMART can fill the missing forecast window from other datasets (by priority) and merge the result without overwriting the primary data.
    • Reduces unnecessary NCSS traffic while keeping the forecast timeline consistent.

  • 🧩 Table-driven dataset selection

    • Dataset handling is now driven by a single metadata table instead of hard-coded rules.
    • Each dataset defines its own:
      • source key,
      • coverage behavior,
      • SMART priority,
      • UI visibility and ordering.
    • Adding new datasets no longer requires touching OptionsFlow logic.

  • 🌍 SILAM Northern Europe v6.1 dataset

    • Added support for the new silam_regional_pollen_v6_1 regional dataset.
    • The new dataset is available for both SMART detection and manual selection.

  • 🔄 Automatic migration of manual dataset selections

    • If Europe v6.0 or Northern Europe v5.9.1 was selected manually, it is automatically replaced during migration:
      • Europe v6.0 → Europe v6.1,
      • Northern Europe v5.9.1 → Northern Europe v6.1.
    • The migration preserves the user’s manual dataset mode instead of switching it to SMART.
    • Existing SMART configurations remain unchanged.
    • The stored base_url migration proposed in PR #23 was extended to all supported legacy dataset replacements instead of applying only to Europe v6.0.
    • This prevents repeated runtime switching when an integration still contains a stored URL for a dataset that has been replaced.

  • 🗑️ Legacy datasets retired

    • The following legacy datasets have been disabled:
      • silam_europe_pollen_v6_0,
      • silam_regional_pollen_v5_9_1.
    • They no longer participate in SMART detection.
    • They are no longer available for manual selection in Options Flow.
    • Their metadata is retained internally only to support migration of existing Config Entries.

  • 🩺 SILAM service availability check during setup

    • Config Flow now starts a background availability check when a new integration is added.
    • The check verifies:
      • that the root THREDDS catalog responds successfully,
      • that the SSL certificate is valid,
      • that the catalog name is FMI SILAM CATALOG,
      • that pollen-related catalogs are available.
    • The setup form displays a compact service status while detailed results remain available in DEBUG logging.

  • 🌐 Localized setup errors

    • Added localized messages for:
      • SSL connection failures,
      • network, DNS, timeout, and server errors,
      • invalid or unavailable SILAM catalog data,
      • locations outside the SILAM coverage area,
      • failures while checking data availability for the selected location.
    • Raw SILAM responses are no longer displayed directly to the user.

  • 🗣️ New and updated translations

  • 🎛️ Unified Options Flow logic

    • Manual dataset selection is now built dynamically from dataset metadata.
    • Availability checks and ordering are handled automatically.
    • SMART and manual modes now share a single, consistent decision model.

  • 🧾 Lean diagnostics output

    • Debug logging no longer prints full merged payloads.
    • Coordinator emits a focused diag block that is easier to read and safer to keep enabled.

🔄 Changes

  • Extended the shared, centralized approach from runs catalog handling to dataset selection logic and runtime dataset health.
  • Added coverage-aware SMART checks that can trigger fallback selection when the current dataset cannot meet the expected forecast window.
  • Introduced a “preferred-first” recovery path that switches back only when a new run is confirmed in the catalog.
  • Reduced repeated SMART probing via cached coordinate-coverage checks.
  • Added lightweight performance markers to simplify troubleshooting of slow fetch cycles.
  • Added the new Northern Europe v6.1 regional dataset to the centralized dataset metadata table.
  • Updated SILAM variable resolution so all supported v6.1 datasets use cnc_POLLEN_OLIVE_m20 for olive pollen data.
  • Added Config Entry migration for manually selected Europe v6.0 and Northern Europe v5.9.1 datasets.
  • Extended stored base_url migration to all supported legacy dataset replacements based on the solution proposed in PR #23.
  • Disabled Europe v6.0 and Northern Europe v5.9.1 in both SMART detection and manual dataset selection.
  • Added a background SILAM service availability check to Config Flow.
  • Added localized service and location validation errors.
  • Added Polish translation support from issue #24.
  • Updated the Slovak translation through PR #27.
  • Improved internal consistency between:
    • dataset probing,
    • manual selection,
    • effective dataset reporting,
    • incremental rebuild vs. network fetch.

🐞 Fixes / Stability

  • Improved resilience during catalog/run transitions by avoiding aggressive switching on short-lived NCSS instability.
  • Prevented noisy logs by emitting only diagnostics instead of full merged structures.
  • Reduced the risk of forecast gaps by filling the missing forecast window without full refetches.
  • Prevented repeated runtime URL switching after a legacy dataset has been migrated.
  • Prevented retired datasets from being selected automatically or manually.
  • Improved setup feedback when the SILAM service is unavailable or the selected location is outside the supported coverage area.
  • Minor cleanups for readability and long-term maintainability.

GitHub DownloadsDynamic JSON Badge

Full Changelog:
v0.3.2...v0.4.0

What's Changed

Contributors

misa1515
Assets 3
Loading