Joint Disaster Classification and Victim Detection using Multi-Task Learning

This repo provides some of the codes used in my following research projects:

1. Joint Disaster Classification and Victim Detection using Multi-Task Learning
2. An Optimized Multi-Task Learning Model for Disaster Classification and Victim Detection in Federated Learning Environments *
   * Research Work No. 2 is an extension based on Research Work No. 1

Feel free to check out our Research Work 2, which is fully open access and free to download.

Overview of Our Project

We design a Multi-Task model for joint disaster classification and victim detection using Representation Similarity Analysis (RSA). We train the model using both the conventional Centralized Learning (CL) and Federated Learning (FL) methods. We also tried Active Learning (AL) to see how it could help in reducing the labeling workload for disaster dataset. Finally, we use OpenVINO for model optimization and inference optimization.

Representation Similarity Analysis

Representation Similarity Analysis (RSA) is to measure the similarity of the feature maps extracted by two models. The paper suggested that "RSA could beused for deciding different branching out locations for different tasks, depending on their similarity with the representations at different depth of the shared root". Our work exploits RSA to pinpoint the optimal branching location (for the multi-task model).

Federated Learning

Federated learning (FL) is an approach to Machine Learning (ML) or Deep Learning (DL), where a shared global model is trained across many participating clients that keep their training data locally. Some of the popular existing FL framework inclucdes (i) Google's TensorFlow Federated (TFF) and (ii) Intel's Open Federated Learning (OpenFL).

Active Learning

Active Learning (AL) is a special case of semi-supervised machine learning. Citing a quote from Datacamp, "The main hypothesis in active learning is that if a learning algorithm can choose the data it wants to learn from, it can perform better than traditional methods with substantially less data for training."

Novelty

1. Existing studies focus on solving single-task issue of disaster classification and victim detection separately. In contrast, we introduce a MTL model by attaching a disaster classification head model to the backbone of a victim detection model. We employ an efficient mathematical analysis to pinpoint the optimal branching location and to prune the head model.
2. The framework design decouples training of two tasks.
3. Most AL methods advocate uncertainty sampling, which selects the most uncertain samples from the unlabeled data pool to label [22]. Such strategy is ill-suited for disaster dataset, where samples from different classes exhibit high similarity. To enable efficient AL-based FL, we introduce a simple heuristic by combining both uncertainty and diversity samplings.
4. The majority of the research tries to accelerate the inference process without detailing the degree of accuracy loss. In contrast, our measurement outputs are based on open-source and production-ready frameworks to ensure reusability, interoperability, and scalability.

Other Repo(s)

For better readability, I separated the following codes from this repo, as they are not directly related to the AL-based-FL for the Multi-Task model.

1. MobileNetV2 for Disaster Classification
2. Representation Similarity Analysis
3. GradCAM for YOLOv3

Tools

TensorFlow 2.0
Open Federated Learning (OpenFL)

References

YOLOv3 for TF 2.0

Acknowledgement

This work is the output of the ASEAN IVO (http://www.nict.go.jp/en/asean_ivo/index.html) project titled “Context-Aware Disaster Mitigation using Mobile Edge Computing and Wireless Mesh Network” and financially supported by NICT (http://www.nict.go.jp/en/index.html).