[Task Clean-up][Benchmark] Dexterous Part 5/11: Add success-rate support to the benchmark utilities

[hujc7]

Summary

• The environment training benchmark utilities now record the Metrics/success_rate signal emitted by the dexterous tasks alongside reward and episode length, with unit tests.
• Fixes training benchmark discovery to exclude inference-only camera benchmark registrations.
• Split out of the lumped validation branch [DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (split into Parts 1-11) #6324 (Part 5 of 11).

Dependencies

• None. (The Metrics/success_rate key it consumes is introduced by [Task Clean-up] Dexterous Part 3/11: Add success-rate metrics to the reorientation Direct tasks #6413 / [Task Clean-up] Dexterous Part 4/11: Enable RSL-RL training for the handover Direct task #6414; recording is a no-op for tasks that do not emit it.)

[greptile-apps]

Greptile Summary

This PR extends the training benchmark utilities to record Metrics/success_rate alongside reward and episode length, using a new sustained-window threshold type (consecutive_samples) that requires N consecutive samples to meet a criterion. It also replaces the narrow endswith(\"Play-v0\") task filter with _is_training_task, which correctly excludes all Play and Benchmark variants.

• env_benchmark_test_utils.py: adds _is_training_task, _parse_threshold_spec, _extract_sustained_feature, and wires success_rate into _extract_log_val; evaluate_job now populates the KPI payload even for missing/non-finite metrics instead of silently continuing.
• test_env_benchmark_test_utils.py: new unit-test file with 9 parametrised scenarios covering the filter, scalar-threshold backward-compatibility, sustained-pass/fail paths, and unsupported-workflow fallback.
• test_environments_training.py: single-line swap to the new _is_training_task helper.

Confidence Score: 4/5

Safe to merge; the changes are additive and record-only for tasks that do not emit success_rate, with no impact on existing reward/episode_length benchmarks

The global non-finite check in _extract_sustained_feature could cause a valid training run to fail its benchmark if any NaN appears in the early success_rate history before the metric stabilises, and the undocumented aggregation switch for reward+consecutive_samples could mislead future config authors. Neither affects existing tasks today but they are latent correctness traps in new code paths.

env_benchmark_test_utils.py — specifically _extract_sustained_feature (global NaN check) and the _extract_log_val reward branch when consecutive_samples is set

Important Files Changed

Filename	Overview
source/isaaclab_tasks/test/benchmarking/env_benchmark_test_utils.py	Core utility extended to add success_rate metric support and sustained-window evaluation via _extract_sustained_feature and _parse_threshold_spec; two subtle design decisions worth attention
source/isaaclab_tasks/test/benchmarking/test_env_benchmark_test_utils.py	New unit-test file covering success_rate evaluation paths, filter logic, and scalar-threshold backward compatibility; missing one edge-case scenario (early NaN warm-up with later valid window)
source/isaaclab_tasks/test/benchmarking/test_environments_training.py	Single-line change replaces the narrow Play-v0 suffix guard with the new _is_training_task helper for broader Benchmark/Play exclusion; change is correct and safe
source/isaaclab_tasks/changelog.d/task-cleanup-dex-part05.rst	New changelog fragment noting the training benchmark discovery fix; content is accurate

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[evaluate_job] --> B[_retrieve_logs]
    B -->|no logs| C[fail: training did not finish]
    B -->|logs found| D[iterate thresholds]
    D --> E[_parse_threshold_spec]
    E -->|plain number| F[consecutive_samples = None]
    E -->|dict| G[consecutive_samples = N]
    F --> H{threshold_name}
    G --> H
    H -->|duration| I[val = duration arg]
    H -->|reward or episode_length or success_rate| J[_extract_log_val]
    J -->|reward, no consecutive| K[_extract_reward avg top-k]
    J -->|other, no consecutive| L[_extract_feature max or min]
    J -->|any metric, with consecutive| M[_extract_sustained_feature]
    M -->|len less than N| N[return None]
    M -->|any non-finite globally| O[return NaN]
    M -->|all finite| P[best window extremum]
    K --> Q{val None or non-finite?}
    L --> Q
    P --> Q
    N --> Q
    O --> Q
    I --> Q
    Q -->|yes| R[payload = None, success = False]
    Q -->|no| S{exceeds threshold?}
    S -->|yes| T[success = False with msg]
    S -->|no| U[payload = val, success unchanged]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[evaluate_job] --> B[_retrieve_logs]
    B -->|no logs| C[fail: training did not finish]
    B -->|logs found| D[iterate thresholds]
    D --> E[_parse_threshold_spec]
    E -->|plain number| F[consecutive_samples = None]
    E -->|dict| G[consecutive_samples = N]
    F --> H{threshold_name}
    G --> H
    H -->|duration| I[val = duration arg]
    H -->|reward or episode_length or success_rate| J[_extract_log_val]
    J -->|reward, no consecutive| K[_extract_reward avg top-k]
    J -->|other, no consecutive| L[_extract_feature max or min]
    J -->|any metric, with consecutive| M[_extract_sustained_feature]
    M -->|len less than N| N[return None]
    M -->|any non-finite globally| O[return NaN]
    M -->|all finite| P[best window extremum]
    K --> Q{val None or non-finite?}
    L --> Q
    P --> Q
    N --> Q
    O --> Q
    I --> Q
    Q -->|yes| R[payload = None, success = False]
    Q -->|no| S{exceeds threshold?}
    S -->|yes| T[success = False with msg]
    S -->|no| U[payload = val, success unchanged]

Loading

_{Reviews (1): Last reviewed commit: "Add success-rate support to the benchmar..." | Re-trigger Greptile}

[greptile-apps]

Global NaN check rejects series with any early non-finite value

np.all(np.isfinite(values)) is applied to the full historical series before any windowing, so a single NaN anywhere — including in early warm-up steps before the metric stabilises — causes the function to return math.nan and fail the threshold even when a valid consecutive_samples-length window exists entirely within the finite portion of the data. For success_rate in particular, early episodes may not record any successes and could produce NaN values before converging; those early samples would silently disqualify an otherwise passing run. Restricting the finiteness check to each candidate window (or filtering non-finite entries before slicing) would make the "best sustained window" semantics consistent with the function's name.

[greptile-apps]

reward bypasses _extract_reward when consecutive_samples is set

When name == "reward" and consecutive_samples is not None, the early return _extract_reward(...) branch is skipped (lines 274–275) and the code falls through to the generic if tag: block, where _extract_sustained_feature is called instead. _extract_reward uses an "average of the top-k" aggregation that is quite different from the sliding-window min/max in _extract_sustained_feature. Any future YAML config that specifies a structured threshold for the reward metric would silently receive a semantically different aggregation than the scalar-threshold path. There are no tests covering this combined path, and the behaviour change is not documented.