[Evaluation] Object-Soft Safety Success is not reproducible with the released threshold map

[j-mayo]

Hello,
thank you for releasing SoftVTBench and the full evaluation pipeline.

While reproducing the soft-suite evaluation from a fresh download of the current Hugging Face dataset, I found that the released Object-Soft asset set does not match configs/safety_thresholds.json. This leaves 100/500 Object-Soft episodes without a deformation threshold and prevents the reported Safety Success metric from being reproduced as specified.

Environment

• SoftVTBench commit: 58056111f01e05bf1a4ae1dee75db4e3d9e7c5be (v0.1 release)
• Dataset: current Arthur12137/SoftVTBench Hugging Face release, freshly downloaded
• Evaluator: experiments/common/softvtbench_metrics.py

Reproduction

python experiments/common/softvtbench_metrics.py \
  /path/to/SoftVTBench_data/object-soft \
  --thresholds configs/safety_thresholds.json \
  --output-dir /tmp/object-soft-metrics \
  --strict

The command exits with:

strict metric validation failed: goal_known=500/500, safe_unknown=100

The 100 unknown episodes are exactly:

50 soft_pastry004,missing_tau:soft_pastry004
50 soft_pastry006,missing_tau:soft_pastry006

Released asset/threshold mismatch

The freshly downloaded Object-Soft HDF5 files contain these 10 assets (50 demonstrations each):

soft_pastry001
soft_pastry002
soft_pastry003
soft_pastry004
soft_pastry005
soft_pastry006
soft_pastry010
soft_stw_sphere_hq
soft_stw_cube_hq
soft_stw_cylinder_hq

However, the released threshold map contains:

soft_pastry001
soft_pastry002
soft_pastry003
soft_pastry005
soft_pastry008
soft_pastry010
soft_pastry011
soft_stw_sphere_hq
soft_stw_cube_hq
soft_stw_cylinder_hq

Thus, soft_pastry004 and soft_pastry006 are missing, while soft_pastry008 and soft_pastry011 are not used by either released soft suite (spatial-soft uses only soft_pastry005).

Without --strict, the scorer treats episodes with a missing tau as safety-unknown and counts them as not safety-successful. Consequently, the released Object-Soft setup can silently depress Safety Success rather than failing before a long closed-loop evaluation. The current preflight checks only that the threshold map is non-empty, and the public-contract test only checks that it has 10 entries, so both checks pass despite the key mismatch.

Relevant files:

• configs/safety_thresholds.json
• experiments/common/softvtbench_metrics.py
• tools/doctor.py
• tests/test_public_contract.py

Related protocol clarifications

1. How should NoDrop_episode be computed in the public evaluator?

Equation 4 in the paper defines:

Safety Success = Goal Success AND NoDrop_episode AND (D_peak <= tau_o)

The paper further states that any transient drop, workspace escape, or loss of stable containment violates NoDrop_episode. The public benchmark_protocol_v1.json and softvtbench_metrics.py, however, currently define Safety Success as only:

Goal Success AND (D_peak <= tau_o)

The environment setup also disables object_dropped termination conditions, and I could not find a per-episode drop flag accumulated by the public scorer. Is NoDrop intended to be inferred from another released signal, or was the paper's NoDrop evaluator not included in v0.1? If the reported paper numbers use an additional rule or private field, could that implementation/definition be released?

2. What is the intended reference for tactile marker motion?

The released converter uses the first timestep's init marker positions as the fixed episode reference (init_pos[0]), while online inference takes the sensor-provided init positions from the first observation after reset. Could you confirm whether the intended reference is:

1. the GelSight undeformed/calibration reference,
2. the first observation of each episode after scene reset, or
3. another collection-time reference?

The paper describes a marker-motion history but does not appear to specify this initial-reference convention, so an explicit statement would help ensure train/eval equivalence.

Requested resolution

Could you please:

1. release the calibrated tau values (and ideally compression-sweep rows) for soft_pastry004 and soft_pastry006, or clarify any intended alias mapping;
2. clarify why soft_pastry008 and soft_pastry011 are present although they are unused by the current released suites;
3. make the preflight/evaluator validate threshold keys against the selected suite's actual asset IDs and fail early on missing keys;
4. clarify/release the NoDrop_episode computation used for the paper's Safety Success results; and
5. document the tactile marker initial-reference convention used during training and evaluation?

Thanks!