PLA

This is the official code for our ICLR2023 paper: "Video Scene Graph Generation from Single-Frame Weak Supervision" openreview link

Installation

• First use the environment.yaml to create the basic env.

• Then install this repository scene_graph_benchmark.

Data Preparing

The pretrained detector

Download from this repository VinVL.

The Action Genome dataset

Download from this repository ActionGenome.

The bboxes and corresponding features of each frame in AG

Use lib/extract_bbox_features.py to extract these features.

The weak-annotation

• (1) Annotation which keeps objects with confidence greater than 0.2. (here)
• (2) From Annotation (1), only keeps the middle frame for each video. (here)
• (3) From Annotation (2), annotated by the model-free strategy with $\eta=0.5$. (here)
• You can also assign annotation with different hyperparemeter by lib/genarate_predicate_pseudo_label.py.

Train

python train.py --cfg demo.yml

Test

python test.py --cfg demo.yml

Model weights of PLA

• Model trained by the middle frame (here)
• Final model (here)

Citations

Please consider citing this project in your publications if it helps your research.

@inproceedings{chen2023video,
  title={Video scene graph generation from single-frame weak supervision},
  author={Chen, Siqi and Xiao, Jun and Chen, Long},
  booktitle={The Eleventh International Conference on Learning Representations},
  year={2023}
}