v0.20.0

Added

• Multi-source create_benchmark(). Members can now come from any combination of item_ids, (dataset_id, ref_id) items, one or more slices (slice_id / slice_ids), and one or more datasets (dataset_id / dataset_ids) — unioned and de-duplicated server-side. At least one source is required (previously exactly one).
• Benchmarks. Full support for benchmark-paradigm evaluation: NucleusClient.create_benchmark() (members from item_ids, (dataset_id, ref_id) items pairs, a slice_id, or a dataset_id; membership frozen at creation), list_benchmarks(), get_benchmark(), update_benchmark(), delete_benchmark(), and list_benchmark_items(), plus the new Benchmark resource with refresh() / update() / delete() / items() / create_evaluation_v2().
• Benchmark evaluations. create_benchmark_evaluation_v2(benchmark_id, model_run_id, ...) evaluates a model run against every benchmark item (uncovered items score as false negatives, keeping leaderboard scores comparable). Accepts rollup_groups, legacy allowed_label_matches / allowed_label_matches_id, exclusion_rules, and preset. Benchmark evaluations are the only creation surface — dataset/slice-scoped evaluation creation is deprecated platform-wide and was never shipped in this SDK.
• Rollup groups. The new RollupGroup type (class_name + labels) is the primary label configuration: each group evaluates a set of raw labels as one class. Presets support it end to end — create_evaluation_v2_preset() / update_evaluation_v2_preset() accept rollup_groups (mutually exclusive with allowed_label_matches), and EvaluationV2Preset exposes the field.
• Exclusion rules. MetadataExclusionRule, LabelExclusionRule, and BoxAreaExclusionRule (or equivalent dicts) drop items/annotations before metrics are computed, passed via exclusion_rules on benchmark evaluation create and presets. EvaluationV2 exposes benchmark_id, rollup_groups, slice_id, exclusion_rules, and exclusion_stats.
• Evaluation V2 presets. Save and reuse evaluation configurations (name + label configuration + exclusion_rules) via list_evaluation_v2_presets(), create_evaluation_v2_preset(), update_evaluation_v2_preset(), and delete_evaluation_v2_preset(), plus the EvaluationV2Preset resource (with update() / delete()). Passing preset= to create_benchmark_evaluation_v2 seeds the label configuration and rules (explicit arguments override the preset's values).
• Results. EvaluationV2.charts() (mAP summary, per-class AP, confusion matrix, PR/F1 curves, TIDE attribution, AP by size) and EvaluationV2.examples() (paginated TP/FP/FN match rows; match_type optional) with EvaluationV2FilterArgs filtering (confidence/IoU ranges, labels, metadata predicates, gt_area_range, slice_ids).
• Cancel & retry. EvaluationV2.cancel() stops a running evaluation; EvaluationV2.retry() re-runs a failed one, reusing its configuration.
• Benchmark leaderboards. leaderboard_ranking(metric_type, benchmark_ids, ...) ranks model runs on one or more benchmarks (metrics: MAP_50, MAP_50_95, AP_SMALL, AP_MEDIUM, AP_LARGE, PRECISION, RECALL, F1; scope / collapse controls), and leaderboard_f1_curve(benchmark_ids, ...) returns F1-vs-confidence curves for the top runs. Requires a Nucleus deployment with leaderboard support.
• Filter schema discovery. EvaluationV2.filter_schema() / NucleusClient.get_evaluation_v2_filter_schema() return the evaluation's filter vocabulary (gt_labels, pred_labels, and item-metadata fields with inferred value types) — the valid inputs for EvaluationV2FilterArgs. Requires the same Nucleus deployment as the leaderboard methods.
• Dataset.evaluation_label_schema() returns the dataset's ground-truth and prediction label vocabularies (gt_labels / prediction_labels) for building rollup groups, label matches, and label exclusion rules.

Changed

• create_benchmark() is now asynchronous. The server creates the benchmark in a "building" state and streams its members in via a background job (removing the previous item-count ceiling on slice/dataset-sourced benchmarks). create_benchmark() blocks on that job by default and returns the completed "ready" benchmark — the return type is unchanged, so existing blocking callers are unaffected. Pass wait_for_completion=False to return the "building" benchmark immediately and poll it yourself via Benchmark.refresh() (checking the new Benchmark.status field). A failed build job raises JobError. Benchmark now exposes status ("building" / "ready" / "failed").