Home

CARE-SIU Wiki

Context-Aware, Reliability-Calibrated, and Explainable Multimodal Social-Interaction Understanding for Safety-Critical Environments

CARE-SIU studies temporal, multimodal, reliability-aware social-interaction understanding under uncertainty, missing evidence, corruption, and domain shift.

---

Project map

Page	Purpose
Architecture	End-to-end model and experiment architecture
Results	Validated quantitative results and negative findings
Datasets-and-Protocols	RWF-2000, synthetic data, splits, and evaluation rules
Training-and-Evaluation	Training, metrics, calibration, and testing discipline
Experiments-and-Reproducibility	Experiment families, seeds, output structure, reproducibility
Research-Integrity-and-Leakage-Audit	Synthetic leakage finding and corrected protocol
Repository-Guide	Source-code and script map

---

Current evidence snapshot

Result	Value
Temporal R3D-18 real-only macro-F1	0.8134 ± 0.0121
Temporal synthetic→real macro-F1	0.8062 ± 0.0080
Transfer mean delta	−0.0072
Paired t-test	p = 0.4383
Wilcoxon	p = 0.6250
Real-only AUROC	~0.8940
Real-only ECE	~0.1237
Transfer ECE	~0.1164

The transfer experiment therefore does not support a classification improvement, although calibration improved slightly.

---

Research-integrity status

The original CARE-Synth-XL dataset was audited after implausibly high modality-ablation scores.

The audit found:

• 10,000 rows;
• only 54 unique pose contents;
• only 50 unique trajectory contents;
• all 54 pose templates crossed train/validation/test;
• all 50 trajectory templates crossed train/validation/test;
• pose-template identity uniquely identified the label.

The affected synthetic ablation results were invalidated for generalization claims.

This Wiki deliberately separates:

implemented system capability
from
experimentally validated scientific evidence.

---

Main architecture

flowchart LR
    R["RGB video"] --> RE["Temporal RGB encoder<br/>R3D-18"]
    A["Audio"] --> AE["Audio encoder"]
    P["Pose"] --> PE["Pose encoder"]
    T["Trajectory"] --> TE["Trajectory encoder"]

    RE --> F["Reliability-aware fusion"]
    AE --> F
    PE --> F
    TE --> F

    Q["Quality / corruption / missingness"] --> C["Reliability estimator"]
    C --> F

    F --> H["Temporal interaction reasoning"]
    H --> Y["Prediction"]
    H --> U["Uncertainty / calibration"]
    H --> X["Explanation / evidence"]

Loading

Go to Architecture for the detailed path and component status.