Skip to content

hubPredEvalsData 1.1.0

Choose a tag to compare

@annakrystalli annakrystalli released this 03 Jun 13:30
v1.1.0
1d4a7f6

New Features

  • Added optional scale-transformation support to predevals-config.yml via schema version v1.1.0 (#39):
    • transform_defaults — top-level default transform applied to all transformable targets.
    • targets[*].transform — per-target transform that overrides transform_defaults, or the literal false to opt a target out.
    • Supported transform functions: log_shift, sqrt, log1p, log, log10, log2.
  • Configured transforms are now applied during scoring (#40).
  • scores.csv is now emitted in wide format, with transformed-scale metrics as <metric>__<label>-suffixed columns (e.g. wis__log). Setting append: false emits only the suffixed columns.
  • Transform-invariant metrics (interval_coverage_<n> and bias) are now always reported on a single, un-suffixed scale in scores.csv and predevals-options.json, even when a transform is configured. Both are unchanged by any strictly monotonic transform (interval containment and the sign of forecast-vs-observation error are rank-based), so the transformed-scale column was a duplicate of the natural-scale column. Matches the invariance set documented in hubverse-org/hubDocs#464 (#63).
  • Existing v1.0.1 configs continue to validate against v1.1.0 without changes.
  • Added generate_predevals_options(), which assembles the contents of the predevals-options.json file used to initialise the predevals dashboard. It returns the validated config with each target's metrics expanded to the columns present in scores.csv (relative-skill metrics, plus transformed-scale <metric>__<label> metrics when a transform applies) and a resolved transform block attached (#41, closes #4).
  • generate_eval_data() now discovers oracle output from hub_path via hubData::connect_target_oracle_output() when oracle_output is not supplied; the argument remains supported for back-compat (#51).

Bug Fixes

  • generate_eval_data() no longer fails on ordinal pmf targets that request rps. The ordinal level order is now read from the hub's tasks.json and forwarded to hubEvals::score_model_out() so scoringutils dispatches the data as ordinal (#48).
  • read_predevals_config() now warns when an ordinal-only pmf metric (e.g. rps) is requested against a pre-v4 tasks-schema (where output_type_id is split across required/optional), and errors if the hub's pmf output_type_id$optional is non-empty (#48).