Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection

🔥 Our Framework

TL;DR: In this repo, we provide the implementation of Multimodal Federated Learning with Joint Modality and Client Selection (MFedMC), a novel methodology for multimodal federated learning that decouples traditional holistic fusion approaches into separate global modality encoders and local fusion modules.

🖥️ Prerequisites

Install the required packages via:

pip install -r requirements.txt

Alternatively, ensure the following dependencies are installed:

python == 3.10.12
torch == 2.6.0
numpy == 1.26.4
scikit-learn == 1.5.1
shap == 0.42.1
h5py == 3.9.0
shap == 0.42.1

🗂️ Folder Structure

MFedMC/
│   README.md
│   requirements.txt
│
├─── ActionSense/
│   │   main.py
│   │   dataset.py
│   │   federated.py
│   │   models.py
│   │   options.py
│   │   utils.py
│
│   # other datasets

• ActionSense/: Core modules for the ActionSense dataset framework. Note: different datasets might have different network architectures and data loaders, so they are separated by folder.
  • main.py: Entry point for initializing data, generalized global modality encoders, and running the MFedMC federated learning loop.
  • dataset.py: Data loading, restructuring, and partitioning utilities (e.g., Dirichlet, Stratified) to handle dataset and modality heterogeneities.
  • federated.py: Core MFedMC algorithm, including local training of modality encoders, personalized local fusion module training, computation of Shapley values, and the joint modality and client selection strategies based on priority metrics.
  • models.py: PyTorch neural network definitions for modality encoders (e.g., Eye, EMG, Tactile, IMU).
  • options.py: Hyperparameters and configuration settings.
  • utils.py: Helper functions for math, normalizations, and SHAP value aggregations.

🏃‍♂ Run Code

Run our framework with the following command:

cd ActionSense
python main.py