SGCap: Decoding Semantic Group For Zero-shot Video Captioning

This code is the official PyTorch implementation of the paper: SGCap: Decoding Semantic Group For Zero-shot Video Captioning

Introduction

SGCap, a method tailored for the zero-shot video captioning task, which leverages a Semantic Group Decoding (SGD) strategy. SGD enables training and inference based on semantic groups, allowing the model to fully exploit multi-frame information. Additionally, we introduce a Key Sentences Selection (KSS) module and a Probability Sampling Supervision (PSS) strategy, encouraging the model to acquire more diverse linguistic knowledge from the sentences.

Quickstart

Important

We conduct experiment on Python 3.13, PyTorch 2.6.0, Transformers 4.48.2, and CUDA 12.0+

1. Requirements

We use conda to control Python virtual environment. Install the dependencies with the following command:

conda create -n GCap python=3.13 # if using conda to control virtual environment
pip install -r requirements.txt

pip install ftfy regex tqdm # to install openai-clip
pip install git+https://github.com/openai/CLIP.git

1. Data preparation

Download the dataset from the URL provided in the paper. For MSR-VTT, MSVD, and VATEX. Place the downloaded data under the folder ./data/origin.

cd scripts

bash data_msvd.sh
bash data_msrvtt.sh
bash data_vatex.sh

The processed data are saved to ./data/extract/

3. Train and evaluate model

• To see the model structure of SGCap, click here.
• The configuration of SGCap is here

Implement the dataset name (msvd, msrvtt, vatex) that you want to train in Line 24 "export DATASET_NAME="name"".

Run the following command to start training process:

bash train.sh

Note

1. Some extra packages of the nltk toolkit may need to be download manually. Please follow the instructions of the error information if reported.
2. We use a 4-D casual mask for the language model which is not compatible with the code in transformers. Please comment the Line 825 "attention_mask = attention_mask.view(batch_size, -1) if attention_mask is not None else None" in transformers/models/gpt2/modeling_gpt2.py (if reporting an error during training)
3. We use Wandb to trace the training process. If you wish to use other logging tools, modify Line 72 "--report_to" in scripts/train.sh to use other tookits.

Download pycocoevalcap toolkit and place it under eval/pycocoevalcap, To evaluate the checkpoint, run:

python evaluate_caption.py \
    --checkpoint "path_to_checkpoint" \
    --dataset_name "dataset name" \
    --caption_from "dataset name"

Results

The evaluation scores and generated captions are in results.

Citation

If you find this repo useful, please cite the following paper.

@article{SGCap,
  author    = {},
  title     = {},
  journal   = {},
  volume    = {},
  pages     = {},
  year      = {2025}
}

Contact

If you have any questions, please contact via email at panzeyucs@hdu.edu.cn