Experiments and Reproducibility

Experiments and Reproducibility

Experiment families

A. Real-only temporal benchmark

Primary validated experiment.

B. Synthetic→real temporal transfer

Paired against the real-only temporal benchmark across identical seed settings.

C. Multimodal ablation

RGB / audio / pose / trajectory combinations.

Status: legacy synthetic results invalidated; must be rerun on leakage-controlled v2.

D. Calibration

Confidence and expected calibration error.

E. Explanations

Sample-level evidence outputs and aggregate explanation summaries.

F. Federated learning

Client-partitioned experimental branch.

G. Continual learning

Round-ordered experimental branch.

H. Edge deployment

TorchScript/export and runtime benchmarking.

---

Required experiment record

Every final run should preserve:

experiment_id
date_time
git_commit
dataset_version
manifest
split_protocol
configuration
random_seed
hardware
software_versions
training_time
checkpoint
validation_metrics
test_metrics
predictions
failure_notes

---

Current automated runner

.\run_care_siu_full.ps1

The runner supports completion markers so completed experiment stages can be skipped and incomplete stages resumed.

---

Consolidated outputs

outputs/
├── temporal/
├── seeds/
├── ablations/
├── federated_multiclass/
├── continual_multiclass/
├── edge_multiclass/
├── explanations_full/
└── autopilot/
    ├── status.csv
    ├── logs/
    └── final_report/

---

Reproducibility rule

A result is not final unless another researcher can determine:

• exactly what data were used;
• how the split was constructed;
• which code/configuration produced the run;
• which seed was used;
• which checkpoint was evaluated;
• why any invalidated runs were excluded.