Task-Aware Asynchronous MTL with Class Incremental Contrastive Learning for Surgical Scene Understanding

Lalithkumar Seenivasan, Mobarakol Islam, Mengya Xu, Chwee Ming Lim, Hongliang Ren

---

Manuscript submitted and under review.

---

Feature Extractor

To be added

Scene graph

To be added

Caption

To be added

Domain adaptation

---

Library Prerequisities.

DGL

DGL is a Python package dedicated to deep learning on graphs, built atop existing tensor DL frameworks (e.g. Pytorch, MXNet) and simplifying the implementation of graph-based neural networks.

Prerequisites

• Python 3.6
• Pytorch 1.1.0
• DGL 0.3
• CUDA 10.0
• Ubuntu 16.04

---

Code Overview

In this project, we implement our method using the Pytorch and DGL library and there are three main folders:

• data/: .
• datasets/: Contains the dataset needed to train the network.
• checkpoints/: Conatins trained weights
• evaluation/: Contains utility tools for evaluation.
• Feature_extractor/: Used to extract features from dataset images to train the scene graph and caption network for algorithm 1, stage 3 & 4.
• models/: Contains network models.
• utils/: Contains utility tools used for training and evaluation.

---

Training

Run Training

• Task-Aware MTL Optimization and fine-tuning (Algorithm 1)

  • MTL:
    • TA_aption_train.py (stage 3) (pending code documentation)
    • TA_scene_graph_train.py (stage 4) (pending code documentation)
  • TA_MTL_finetune_train.py (stage 5) (Pending code documentation)

• Vanilla MTL (MTL-V)

  • MTL_V_train.py (pending code documentation)

• Knowledge Distillation-based MTL Optmization (Algorithm 2)

  • MTL_KD_train.py (pending code documentation)

• Knowledge Distillation-based MTL Optmization and fine-tuning

  • MTL_KD_train.py (pending code documentation)
  • MTL_KD_train.py (pending code documentation)

---

Evaluation

Dataset

• Download validation datasetTo_be_released and palce them inside the folder datasets/
• MTL_evaluation.py

checkpoint

• Download checkpoints To_be_released and place them inside the folder checkpoints/

running evaluation

• MTL_evaluation.py
  • Set mtl_version, adapt_type and domain

---

Acknowledgement

Code adopted and modified from :

1. Visual-Semantic Graph Attention Network for Human-Object Interaction Detecion
   • Paper Visual-Semantic Graph Attention Network for Human-Object Interaction Detecion.
   • Official Pytorch implementation code.
2. End-to-End Incremental Learning
   • Paper End-to-End Incremental Learning.
   • Pytorch implementation code.
3. Curriculum by smoothing
   • Paper Curriculum by smoothing.
   • Pytorch implementation code.

---