CAD — Contextual Multi-modal Alignment for Dynamic AVQA

Official PyTorch implementation of:

CAD - Contextual Multi-modal Alignment for Dynamic AVQA Asmar Nadeem, Adrian Hilton, Robert Dawes, Graham Thomas, Armin Mustafa. WACV 2024. [paper]

CAD tackles Audio-Visual Question Answering (AVQA) on dynamic scenes by aligning audio and visual streams on three levels — Spatial, Temporal, and Semantic — and improves the average MUSIC-AVQA accuracy by ~9.4% over the ST-AVQA baseline. The architecture and training scaffolding follow the official ST-AVQA / MUSIC-AVQA codebase (GeWu-Lab/MUSIC-AVQA).

The three contributions

#	Contribution	Level	Code
1	Contextual Block — parameter-free stochastic gating of visual features	Spatial	cad/models/contextual_block.py
2	AV Fine Temporal Alignment — self-supervised time-label pre-training	Temporal	cad/models/pretrain_model.py, cad/pretrain.py
3	Three Cross-Attention Blocks — semantic audio/visual balancing	Semantic	cad/models/cross_attention.py, cad/models/cad_model.py

Forward flow (Algorithms 1 & 2)

t   = QstEncoder(question)                  # GloVe-init LSTM -> [B,1,512]
a   = AudioProjector(panns_feats)           # [B, Ta, 512]
v   = ContextualBlock(VisualProjector(vit)) # spatial gating (train only) -> [B, Tv*N, 512]
a_t = CAB1(q=t,   k=v=a)                     # text-queried audio
v_t = CAB2(q=t,   k=v=v)                     # text-queried visual
v_at= CAB3(q=a_t, k=v=v)                     # audio-queried visual
ans = FC( concat(a_t, v_t, v_at) )           # CrossEntropy AVQA loss (Eq. 2)

Each CAB(q,k,v) = MultiHeadAttention → FC→ReLU→FC→ReLU → residual add → LayerNorm.

Install

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt          # torch, numpy, pandas, einops, pyyaml, pytest

Quick check (no data needed)

pytest -q                  # 24 unit/integration tests
python -m cad.pretrain --smoke   # AVFA pre-training, 1 step on synthetic data
python -m cad.train    --smoke   # fine-tuning, 1 step on synthetic data

Training on real data

CAD consumes cached features (as in ST-AVQA): per-second PANNs audio embeddings and per-frame ViT patch tokens stored as .npy, plus the official MUSIC-AVQA JSON splits.

data/
  json/   avqa-train.json  avqa-val.json  avqa-test.json   # from GeWu-Lab/MUSIC-AVQA
  feats/
    panns/<video_id>.npy        # [60, 2048]
    vit_14x14/<video_id>.npy    # [60, 196, 768]

1. AVFA pre-training on ACAV100M (music category, stitched 60s clips):

   python -m cad.pretrain --manifest data/acav/manifest.json \
       --audio-dir data/acav/panns --visual-dir data/acav/vit \
       --positive-prob 0.6 --epochs 10 --checkpoint cad_avfa

2. Fine-tune on MUSIC-AVQA (init from the pre-trained backbone):

   python -m cad.train --audio-dir data/feats/panns --visual-dir data/feats/vit_14x14 \
       --label-train data/json/avqa-train.json --label-val data/json/avqa-val.json \
       --glove-path data/glove.6B.300d.txt \
       --pretrained-backbone checkpoints/cad_avfa.pt --epochs 25

3. Evaluate (per-category MUSIC-AVQA metrics, 9 question types):

   python -m cad.eval --audio-dir data/feats/panns --visual-dir data/feats/vit_14x14 \
       --label-test data/json/avqa-test.json --checkpoint checkpoints/cad.pt

Default hyper-parameters (paper Section 4.2 + supplementary)

See configs/cad.yaml. Key values: dim 512, 8 heads, Adam, lr 1e-4, batch 64, 25 train / 10 pre-train epochs; Contextual Block feature_select_prob=0.8, mask_prob=0.9, threshold_ratio=0.9; AVFA pair sampling 0.6/0.4 over N=60 one-second cues.

Feature extraction (from raw videos)

feat_script/ extracts CAD's features directly from the MUSIC-AVQA videos (PANNs audio at 32 kHz, ViT-B/16 patch tokens), matching Section 4.2:

# PANNs (Cnn14) audio [60,2048] + ViT-B/16 patch tokens [60,196,768] per video
python feat_script/extract_features.py \
    --video-dir /path/to/MUSIC-AVQA-videos-Real \
    --audio-out data/feats/panns --visual-out data/feats/vit_14x14

# filter the official JSON splits to the videos that have features
python feat_script/build_splits.py --src-json-dir data/json_src \
    --feat-dir data/feats/vit_14x14 --out-dir data/json

Repository layout

cad/
  models/   contextual_block, cross_attention, encoders, cad_model, pretrain_model
  data/     music_avqa, acav_pretrain, synthetic (tests/smoke)
  train.py  pretrain.py  eval.py  config.py  utils.py
feat_script/  extract_features, build_splits
tests/      one file per component (24 tests)
configs/    cad.yaml

Citation

@inproceedings{nadeem2024cad,
  title     = {CAD - Contextual Multi-modal Alignment for Dynamic AVQA},
  author    = {Nadeem, Asmar and Hilton, Adrian and Dawes, Robert and Thomas, Graham and Mustafa, Armin},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2024}
}

License

MIT — see LICENSE.