Skip to content

Add optional direct exceedance-probability output representation #4

Description

@mabesa

Summary

Some hydrological risk models produce direct threshold-exceedance probabilities as a
native forecast product, in addition to or instead of reconstructing those probabilities
from trajectories or quantiles. ForecastInterface currently supports deterministic,
quantile, and trajectory output representations, but it has no first-class way to
return model-native probabilities such as P(discharge > danger_level_3_threshold).

This came up while evaluating Extreme Quantile Regression Neural Networks (EQRN) and
extreme conformal prediction methods for flood-risk forecasting. These methods can
return high conditional quantiles and direct exceedance probabilities from the fitted
tail model. For alerting, the direct probability may be more faithful than deriving a
probability later through coarse quantile interpolation.

Proposal

Add an optional exceedance-probability payload to VariableOutput, alongside the
existing physical forecast payloads:

  • deterministic
  • quantiles
  • trajectories
  • proposed: exceedance_probabilities

The new payload should be additive. Models should still be encouraged to return the
actual forecast variable (discharge, water_level, etc.) as deterministic,
quantiles, or trajectories where possible. Exceedance probabilities should not replace
the physical forecast variable by default, because downstream systems still need the
physical forecast for plotting, quality control, verification, storage, arbitrary later
thresholds, and multi-model combination.

Possible schema:

class ThresholdDirection(Enum):
    ABOVE = "above"
    BELOW = "below"

class ExceedanceProbabilityData(BaseModel):
    data: pl.DataFrame
    # Required columns:
    # - issue_datetime: datetime
    # - datetime: datetime
    # - threshold_value: float
    # - probability: float  # [0, 1]
    # Optional columns:
    # - threshold_name: str
    # - threshold_unit: Unit or string
    # - direction: "above" | "below"
    # - source: "model_native" | "derived_from_quantiles" | "derived_from_trajectories"

Validation should ensure:

  • probabilities are finite and in [0, 1]
  • every row has a threshold value and direction
  • units are compatible with the target variable where unit metadata is available
  • horizon/issue-datetime consistency follows the same rules as deterministic,
    quantile, and trajectory payloads

Open design questions

  1. Should exceedance probabilities be part of VariableOutput for the same target
    variable, or a separate station-level risk output keyed by target variable and
    threshold?
  2. Should the target declaration (TargetSpec.representations) gain a new
    OutputRepresentation.EXCEEDANCE_PROBABILITY, or should this be modeled as
    auxiliary metadata attached to a target that still declares one of the existing
    physical representations?
  3. Should FI permit a model that returns only exceedance probabilities and no physical
    forecast variable? My recommendation is no for the main ForecastModel contract.
    If risk-only outputs are needed, they may deserve a separate RiskForecastModel
    contract so downstream systems do not assume the physical forecast distribution is
    available.
  4. How should threshold identity be represented when the threshold comes from a
    deployment-specific alert configuration rather than the model itself?

Motivation

SAPPHIRE Flow currently derives threshold exceedance probabilities from stored
forecast ensembles:

  • trajectories/members: fraction of members exceeding the threshold
  • quantiles: CDF interpolation between quantile levels

That works for general probabilistic forecasts, but can be suboptimal for tail-focused
models. EQRN-style models estimate the conditional tail directly with a GPD component,
so the probability of exceeding an operational danger threshold can be a native model
output. Preserving that value avoids losing information through a coarse quantile grid
and makes alert verification with Brier score/reliability diagrams more direct.

Compatibility

This can be backward-compatible if:

  • the new payload is optional
  • existing deterministic/quantile/trajectory validators remain unchanged
  • adapters that do not understand exceedance probabilities ignore the payload
  • models that emit only current output types continue to validate exactly as before

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions