[ICASSP 2025] DiffListener: Discrete Diffusion Model for Listener Generation

Abstract

The listener head generation (LHG) task aims to generate natural nonverbal listener responses based on the speaker’s multimodal cues. While prior work either rely on limited modalities (e.g. audio and facial information) or employ autoregressive approaches which have limitations such as accumulating prediction errors. To address these limitations, we propose DiffListener, a discrete diffusion based approach for non-autoregressive listener head generation. Our model takes the speaker’s facial information, audio, and text as inputs, additionally incorporating facial differential information to represent the temporal dynamics of expressions and movements. With this explicit modeling of facial dynamics, DiffListener can generate coherent reaction sequences in a non-autoregressive manner. Through comprehensive experiments, DiffListener demonstrates state-of-the-art performance in both quantitative and qualitative evaluations. The user study shows that DiffListener generates natural context-aware listener reactions that are well synchronized with the speaker.

Env download

We use two environments to train the model:

1. Learning2Listen
2. LM-Listener

Please follow the instructions in the repositories above to set up the environments.

Data preparation

Please refer the previous repo and download the each datasets and place it below the ./data
Learning2Listen
LM-Listener

Data preprocessing

Run the following scripts in order:

1. python segment_to_my_fit.py – segment raw files into our feature format
2. python data_preprocessing.py – preprocess the data
3. python mean_std.py – compute mean and standard deviation
4. python data/test_txt_generate.py – generate test.txt

Checkpoint Download

Download the checkpoint from the Drive
place the trevor_difflistener.pth to ./ckpt
place the trevor_vqvae.pth to ./VQ/ckpt

First Stage, VQ-VAE Training (LM-Listener env)

1. cd ./VQ

2.  python train_vq_trevor.py \
    --batch-size 256 \
    --lr 2e-4 \
    --total-iter 300000 \
    --lr-scheduler 200000 \
    --nb-code 256 \
    --down-t 3 \
    --depth 3 \
    --window-size 32 \
    --dilation-growth-rate 3 \
    --out-dir trevor_vq \
    --dataname face_trevor \
    --vq-act relu \
    --quantizer ema_reset \
    --loss-vel 0.5 \
    --recons-loss l1_smooth \
    --exp-name VQVAE_trevor_ours \
    --eval-split val \
    --data_root ./data/preprocessed_lm_listener_tdmm_data2
   

Second Stage, DiffListener Training (L2L env)

python running_command/run_difflistener_trevor.py

Inference (L2L Env)

python inference_difflistener_trevor.py

Visualize

Please follow the visualization section in LM-Listener.