Multimodal Pretraining & Linear Probe Pipeline

🎉 Accepted to CVPR 2026.
Paper: CVF Open Access
This repository accompanies DETACH, a unified pipeline for multimodal sensor-video action recognition. If it helps your research, please consider starring the repo.

Overall Architecture of DETACH. Stage 1 learns spatial representations, where highlighted samples in the cluster visualization indicate cluster centroids. Stage 2 leverages these spatial features to guide temporal alignment.

This DETACH repository provides a unified pipeline for pretraining and linear probing for multimodal sensor-video action recognition.
It supports Opportunity++, HWU-USP, and any dataset formatted into 2-second trimmed windows.

1. Pretraining Usage

Run pretraining:

PROJECT_NAME: Weights & Biases project name used for logging.

python pretrain.py \
    --dataset_name Opportunity++ \
    --model_name detach \
    --project_name PROJECT_NAME \
    --epochs 50 \
    --lr 1e-4 \
    --batch_size 256 \
    --num_frames 20 \
    --embedding_dim 512 \
    --threshold_epoch 5 \
    --centroid_threshold 0.75 \
    --video_classifier_epoch 10 \
    --bad_correction_epoch 10

2. Linear Probe Usage

python linear_probe.py \
    --checkpoint_path /path/to/pretrained.ckpt \
    --linear_epochs 50 \
    --lr 1e-3 \
    --batch_size 256 \
    --embedding_dim 256

3. Pretraining & Linear_probing JSON Format

The same JSON format is used for both pretraining and linear probing.
label is required only for linear probing.

{
   "data": [
       {
           "video_id": "S3-ADL4_000810000_000812000",
           "frame_path": "trim_2s_video/S3-ADL4/...mp4",
           "sensor_path": "trim_2s_imu_last/S3-ADL4/...csv",
           "label": 13 
       }
   ]
}

4. Sequence Lienar_probling JSON Format

Some datasets provide samples grouped by sequence.
Each sequence contains multiple 2-second windows with sensor–video pairs.

Example:

{
    "data": [
        {
            "sequence_id": "tidy_s02",
            "windows": [
                {
                    "video_id": "tidy_s02_0000000000_0000002000",
                    "frame_path": "trim_2s_video/tidy/...mp4",
                    "sensor_path": "trim_2s_sensor/tidy/...csv",
                    "label": 6,
                    "class_name": "tidy",
                },
                {
                    "video_id": "tidy_s02_0000001000_0000003000",
                    "frame_path": "trim_2s_video/tidy/...mp4",
                    "sensor_path": "trim_2s_sensor/tidy/...csv",
                    "label": 6,
                    "class_name": "tidy",
                }     
            ]
        }
    ]
}