Summary
Some hydrological risk models produce direct threshold-exceedance probabilities as a
native forecast product, in addition to or instead of reconstructing those probabilities
from trajectories or quantiles. ForecastInterface currently supports deterministic,
quantile, and trajectory output representations, but it has no first-class way to
return model-native probabilities such as P(discharge > danger_level_3_threshold).
This came up while evaluating Extreme Quantile Regression Neural Networks (EQRN) and
extreme conformal prediction methods for flood-risk forecasting. These methods can
return high conditional quantiles and direct exceedance probabilities from the fitted
tail model. For alerting, the direct probability may be more faithful than deriving a
probability later through coarse quantile interpolation.
Proposal
Add an optional exceedance-probability payload to VariableOutput, alongside the
existing physical forecast payloads:
deterministic
quantiles
trajectories
- proposed:
exceedance_probabilities
The new payload should be additive. Models should still be encouraged to return the
actual forecast variable (discharge, water_level, etc.) as deterministic,
quantiles, or trajectories where possible. Exceedance probabilities should not replace
the physical forecast variable by default, because downstream systems still need the
physical forecast for plotting, quality control, verification, storage, arbitrary later
thresholds, and multi-model combination.
Possible schema:
class ThresholdDirection(Enum):
ABOVE = "above"
BELOW = "below"
class ExceedanceProbabilityData(BaseModel):
data: pl.DataFrame
# Required columns:
# - issue_datetime: datetime
# - datetime: datetime
# - threshold_value: float
# - probability: float # [0, 1]
# Optional columns:
# - threshold_name: str
# - threshold_unit: Unit or string
# - direction: "above" | "below"
# - source: "model_native" | "derived_from_quantiles" | "derived_from_trajectories"
Validation should ensure:
- probabilities are finite and in
[0, 1]
- every row has a threshold value and direction
- units are compatible with the target variable where unit metadata is available
- horizon/issue-datetime consistency follows the same rules as deterministic,
quantile, and trajectory payloads
Open design questions
- Should exceedance probabilities be part of
VariableOutput for the same target
variable, or a separate station-level risk output keyed by target variable and
threshold?
- Should the target declaration (
TargetSpec.representations) gain a new
OutputRepresentation.EXCEEDANCE_PROBABILITY, or should this be modeled as
auxiliary metadata attached to a target that still declares one of the existing
physical representations?
- Should FI permit a model that returns only exceedance probabilities and no physical
forecast variable? My recommendation is no for the main ForecastModel contract.
If risk-only outputs are needed, they may deserve a separate RiskForecastModel
contract so downstream systems do not assume the physical forecast distribution is
available.
- How should threshold identity be represented when the threshold comes from a
deployment-specific alert configuration rather than the model itself?
Motivation
SAPPHIRE Flow currently derives threshold exceedance probabilities from stored
forecast ensembles:
- trajectories/members: fraction of members exceeding the threshold
- quantiles: CDF interpolation between quantile levels
That works for general probabilistic forecasts, but can be suboptimal for tail-focused
models. EQRN-style models estimate the conditional tail directly with a GPD component,
so the probability of exceeding an operational danger threshold can be a native model
output. Preserving that value avoids losing information through a coarse quantile grid
and makes alert verification with Brier score/reliability diagrams more direct.
Compatibility
This can be backward-compatible if:
- the new payload is optional
- existing deterministic/quantile/trajectory validators remain unchanged
- adapters that do not understand exceedance probabilities ignore the payload
- models that emit only current output types continue to validate exactly as before
Summary
Some hydrological risk models produce direct threshold-exceedance probabilities as a
native forecast product, in addition to or instead of reconstructing those probabilities
from trajectories or quantiles. ForecastInterface currently supports deterministic,
quantile, and trajectory output representations, but it has no first-class way to
return model-native probabilities such as
P(discharge > danger_level_3_threshold).This came up while evaluating Extreme Quantile Regression Neural Networks (EQRN) and
extreme conformal prediction methods for flood-risk forecasting. These methods can
return high conditional quantiles and direct exceedance probabilities from the fitted
tail model. For alerting, the direct probability may be more faithful than deriving a
probability later through coarse quantile interpolation.
Proposal
Add an optional exceedance-probability payload to
VariableOutput, alongside theexisting physical forecast payloads:
deterministicquantilestrajectoriesexceedance_probabilitiesThe new payload should be additive. Models should still be encouraged to return the
actual forecast variable (
discharge,water_level, etc.) as deterministic,quantiles, or trajectories where possible. Exceedance probabilities should not replace
the physical forecast variable by default, because downstream systems still need the
physical forecast for plotting, quality control, verification, storage, arbitrary later
thresholds, and multi-model combination.
Possible schema:
Validation should ensure:
[0, 1]quantile, and trajectory payloads
Open design questions
VariableOutputfor the same targetvariable, or a separate station-level risk output keyed by target variable and
threshold?
TargetSpec.representations) gain a newOutputRepresentation.EXCEEDANCE_PROBABILITY, or should this be modeled asauxiliary metadata attached to a target that still declares one of the existing
physical representations?
forecast variable? My recommendation is no for the main
ForecastModelcontract.If risk-only outputs are needed, they may deserve a separate
RiskForecastModelcontract so downstream systems do not assume the physical forecast distribution is
available.
deployment-specific alert configuration rather than the model itself?
Motivation
SAPPHIRE Flow currently derives threshold exceedance probabilities from stored
forecast ensembles:
That works for general probabilistic forecasts, but can be suboptimal for tail-focused
models. EQRN-style models estimate the conditional tail directly with a GPD component,
so the probability of exceeding an operational danger threshold can be a native model
output. Preserving that value avoids losing information through a coarse quantile grid
and makes alert verification with Brier score/reliability diagrams more direct.
Compatibility
This can be backward-compatible if: