Architecture

Architecture

1. System objective

CARE-SIU is designed for social-interaction understanding where the system may observe incomplete, noisy, corrupted, or distribution-shifted evidence.

The design therefore separates four concerns:

1. representation — what each modality observes;
2. reliability — how trustworthy each observation is;
3. reasoning — how evidence is combined over time;
4. decision support — prediction, calibration, explanation, and defer/review behaviour.

---

2. End-to-end architecture

flowchart TB
    subgraph I["A. Input observations"]
        RGB["RGB video<br/>frames / clips"]
        AUD["Audio<br/>waveform / features"]
        POSE["Pose<br/>joint sequences"]
        TRAJ["Trajectory<br/>positions + velocities"]
    end

    subgraph P["B. Preprocessing"]
        VS["Temporal sampling<br/>resize / normalization"]
        AS["Audio preprocessing"]
        PS["Pose normalization<br/>missing-joint handling"]
        TS["Trajectory normalization<br/>tracking noise handling"]
    end

    subgraph E["C. Encoders"]
        R3D["R3D-18<br/>temporal RGB representation"]
        AENC["Audio encoder"]
        PENC["Pose encoder"]
        TENC["Trajectory encoder"]
    end

    subgraph R["D. Reliability"]
        MISS["Modality availability"]
        CORR["Corruption / quality"]
        REL["Reliability scores"]
    end

    subgraph F["E. Fusion & reasoning"]
        FUSE["Reliability-aware<br/>multimodal fusion"]
        TEMP["Temporal interaction<br/>reasoning"]
    end

    subgraph O["F. Outputs"]
        CLASS["Interaction class / risk"]
        CAL["Confidence / calibration"]
        EXPL["Explanation / evidence"]
        DEFER["Human review / deferral"]
    end

    RGB --> VS --> R3D
    AUD --> AS --> AENC
    POSE --> PS --> PENC
    TRAJ --> TS --> TENC

    R3D --> FUSE
    AENC --> FUSE
    PENC --> FUSE
    TENC --> FUSE

    MISS --> REL
    CORR --> REL
    REL --> FUSE

    FUSE --> TEMP
    TEMP --> CLASS
    TEMP --> CAL
    TEMP --> EXPL

    CAL --> DEFER
    REL --> DEFER

Loading

---

3. Validated real-data architecture

The strongest currently validated experiment uses the temporal RGB path:

flowchart LR
    D["RWF-2000"] --> S["Video decoding + sampling"]
    S --> R["R3D-18"]
    R --> Z["Temporal video embedding"]
    Z --> C["Binary classifier"]
    C --> F1["Macro-F1"]
    C --> ROC["AUROC / AUPRC"]
    C --> ECE["Calibration / ECE"]
    C --> CM["Confusion + failure analysis"]

Loading

This path produced the strongest validated CARE-SIU real-data classification result.

---

4. Multimodal branch

The codebase supports the following conceptual representation:

[ z_m = E_m(x_m), \qquad m \in {\text{RGB},\text{audio},\text{pose},\text{trajectory}} ]

where (E_m) is a modality-specific encoder.

A reliability-aware fusion module can be written generically as:

[ z = \sum_m \alpha_m z_m, ]

with normalized reliability weights:

[ \alpha_m = \frac{a_m \exp(r_m)} {\sum_j a_j \exp(r_j)}, ]

where:

• (a_m \in {0,1}) represents modality availability;
• (r_m) represents estimated evidence reliability.

This formulation captures the intended architectural principle: missing or degraded modalities should not contribute equally to high-quality evidence.

---

5. Temporal representation upgrade

The earlier weak/pooled real-video benchmark achieved approximately:

0.6661 ± 0.0067 macro-F1

The R3D-18 temporal representation increased the real-only five-seed result to:

0.8134 ± 0.0121 macro-F1

This is an absolute improvement of approximately:

[ 0.8134 - 0.6661 \approx 0.1473 ]

The result supports the need for temporal video representations in this task.

---

6. Reliability and calibration

CARE-SIU treats prediction confidence as an object of evaluation rather than assuming that the largest softmax probability is trustworthy.

The current temporal runs evaluate:

• expected calibration error (ECE);
• AUROC;
• AUPRC;
• confusion patterns;
• uncertainty behaviour.

The transfer condition slightly reduced average ECE (~0.1237 → ~0.1164) even though it did not improve macro-F1.

---

7. Explanation path

Explanation outputs are treated as an evidence layer:

flowchart LR
    P["Prediction"] --> E["Explanation"]
    Z["Encoded evidence"] --> E
    R["Reliability state"] --> E
    E --> Q["Explanation diagnostics"]
    Q --> F["Faithfulness / consistency / failure review"]

Loading

A visually plausible explanation is not automatically considered faithful. CARE-SIU therefore keeps explanation generation and explanation evaluation conceptually separate.

---

8. Federated, continual, and edge branches

flowchart LR
    BASE["CARE-SIU model"]
    BASE --> FED["Federated branch<br/>client-based partitions"]
    BASE --> CONT["Continual branch<br/>round-ordered learning"]
    BASE --> EDGE["Edge export<br/>TorchScript + latency"]

Loading

These branches are implemented as experimental extensions. They should not be interpreted as primary performance improvements without matched evaluation.

---

9. Architecture-to-evidence matrix

Component	Implemented	Validated on real data	Scientific claim status
Temporal RGB encoder	✅	✅	Supported
Binary real-video classifier	✅	✅	Supported
Calibration analysis	✅	✅	Supported
Synthetic→real temporal transfer	✅	✅	Evaluated; no classification gain
Audio branch	✅	Partial	Pending leakage-controlled evaluation
Pose branch	✅	Partial	Legacy synthetic results invalidated
Trajectory branch	✅	Partial	Legacy synthetic results invalidated
Reliability-aware multimodal fusion	✅	Partial	Requires stronger multimodal validation
Federated branch	✅	Experimental	No primary claim
Continual branch	✅	Experimental	No primary claim
Edge export	✅	Experimental	Deployment feasibility
Explanation branch	✅	Experimental	Further faithfulness validation needed