-
Notifications
You must be signed in to change notification settings - Fork 0
Results
CARE-SIU separates:
- validated real-data results;
- historical baselines;
- negative findings;
- invalidated experiments;
- pending experiments.
This prevents implementation breadth from being confused with scientific evidence.
The main real-data benchmark uses 1,994 readable RWF-2000 videos.
Five random seeds:
1, 7, 21, 42, 84
| Experiment | Macro-F1 ↑ | Status |
|---|---|---|
| Earlier pooled/weak RGB baseline | 0.6661 ± 0.0067 | Historical |
| Temporal R3D-18 real-only | 0.8134 ± 0.0121 | Validated |
| Temporal R3D-18 synthetic→real | 0.8062 ± 0.0080 | Validated transfer experiment |
The temporal real-only model improves over the earlier pooled benchmark by approximately:
+0.1473 absolute macro-F1
| Seed | Real-only | Synthetic→real | Δ |
|---|---|---|---|
| 1 | 0.8225 | 0.8018 | −0.0207 |
| 7 | 0.8274 | 0.7947 | −0.0327 |
| 21 | 0.8023 | 0.8098 | +0.0075 |
| 42 | 0.8150 | 0.8149 | −0.00003 |
| 84 | 0.7998 | 0.8098 | +0.0100 |
Summary:
| Statistic | Value |
|---|---|
| Real-only mean | 0.8134 |
| Real-only SD | 0.0121 |
| Transfer mean | 0.8062 |
| Transfer SD | 0.0080 |
| Mean Δ | −0.0072 |
| Median Δ | ≈ −0.00003 |
| Improved seeds | 2 / 5 |
| Test | Statistic | p-value |
|---|---|---|
| Paired t-test | −0.8599 | 0.4383 |
| Wilcoxon signed-rank | 5.0 | 0.6250 |
The current temporal experiment does not support the hypothesis that synthetic pretraining improves real-world classification performance.
This negative result is retained because it constrains the claim and informs the redesign of the synthetic-data pipeline.
Approximate aggregate metrics from the completed temporal runs:
| Metric | Real-only | Synthetic→real | Better |
|---|---|---|---|
| ECE ↓ | ~0.1237 | ~0.1164 | Transfer |
| AUROC ↑ | ~0.8940 | ~0.8901 | Real-only |
| AUPRC ↑ | ~0.900 | ~0.900 | Comparable |
The transfer condition therefore shows a small calibration improvement without a macro-F1 improvement.
Before the temporal R3D-18 upgrade, the five-seed pooled/weak experiment produced:
| Condition | Macro-F1 |
|---|---|
| Real-only | 0.6661 ± 0.0067 |
| Synthetic→real | 0.6574 ± 0.0252 |
| Mean Δ | −0.0087 |
Statistical tests:
- paired t-test: p = 0.463470
- Wilcoxon: p = 0.437500
The temporal encoder substantially improved the real-data representation quality, but synthetic pretraining still did not become a supported classification improvement.
The corrected-path synthetic ablation rerun produced near-perfect results for several modalities, including pose-only and trajectory-only.
Those scores are not valid experimental evidence.
| Finding | Value |
|---|---|
| Dataset rows | 10,000 |
| Classes | 24 |
| Unique pose contents | 54 |
| Unique trajectory contents | 50 |
| Combined pose/trajectory templates | 54 |
| Pose templates in multiple splits | 54 |
| Trajectory templates in multiple splits | 50 |
| Pose templates used by multiple labels | 0 |
| Rows involved in pose cross-split duplication | 10,000 |
Most labels contained only two independent pose templates.
Therefore the model could identify a class-specific template already seen in training.
Synthetic ablation performance:
INVALID FOR GENERALIZATION CLAIMS
The affected results should be retained only as an audit trail.
- temporal video modeling substantially improves the real-only RWF-2000 benchmark relative to the earlier pooled representation;
- five-seed temporal performance is stable around macro-F1 0.81;
- synthetic pretraining does not significantly improve temporal real-world classification under the completed protocol;
- calibration shows a small transfer-related improvement;
- the original synthetic generator contained severe cross-split template leakage, discovered through content hashing and template analysis.
- general multimodal superiority over RGB on real-world data;
- improved real classification from synthetic pretraining;
- state-of-the-art claims based on the invalidated synthetic ablations;
- external cross-dataset generalization;
- deployment-level safety guarantees.
- Generate leakage-controlled CARE-Synth-XL-v2.
- Require template-disjoint train/validation/test splits.
- Re-run the 45 modality ablations.
- Add a second independent real dataset.
- Evaluate missing-modality and corruption robustness.
- Quantify selective prediction / review coverage.
- Strengthen explanation faithfulness evaluation.