Awesome-Foundation-Models-with-Federated-Learning

This repository is primarily based on our survey paper:

Synergizing Foundation Models and Federated Learning: A Survey

Shenghui Li, Fanghua Ye, Meng Fang, Jiaxu Zhao,

Yun-Hin Chan, Edith C.-H. Ngai, Thiemo Voigt

If you find this survey useful for your research, please consider citing:

@misc{li2024synergizing,
      title={Synergizing Foundation Models and Federated Learning: A Survey},
      author={Shenghui Li and Fanghua Ye and Meng Fang and Jiaxu Zhao and Yun-Hin Chan and Edith C. -H. Ngai and Thiemo Voigt},
      year={2024},
      eprint={2406.12844},
      archivePrefix={arXiv}
}

Table of Contents

Adaptability
- Domain-Centric Adaptation
  - Multi-Domain Adaptation
- Client-Centric Adaptation
Efficiency
Trustworthiness
Application

Adaptability

Domain-Centric Adaptation

Multi-Domain Adaptation

Title	Venue	Year
Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization	TASLP	2024-06
DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning	CVPR	2024-06
Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning	AAAI	2024-03

Client-Centric Adaptation

Title	Venue	Year	GitHub
FedSelect: Customized Selection of Parameters for Fine-Tuning during Personalized Federated Learning	CVPR	2024-06
FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning	IEEE DEB	2023-03

Efficiency

Knowledge Distillation

Title	Venue	Year	GitHub
FedPFT: Federated Proxy Fine-Tuning of Foundation Models	IJCAI	2024-08
FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning	AAAI	2024-03

Selective Tuning

Title	Venue	Year
FedPFT: Federated Proxy Fine-Tuning of Foundation Models	IJCAI	2024-08
FedSelect: Customized Selection of Parameters for Fine-Tuning during Personalized Federated Learning	CVPR	2024-06
Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation	WWW	2024-05

Additive Tuning

Adapter Tuning

Title	Venue	Year
Federated Adaptation for Foundation Model-based Recommendations	IJCAI	2024-08
Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization	TASLP	2024-06
Learning Task-Specific Initialization for Effective Federated Continual Fine-Tuning of Foundation Model Adapters	IEEE CAI	2024-06
FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning	IEEE DEB	2023-03

Prompt Tuning

Textual Prompt Tuning

Title	Venue	Year
FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models	ICML	2024-07
Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning	NAACL	2024-06
Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization	EMNLP	2023-12
Tunable Soft Prompts are Messengers in Federated Learning	EMNLP	2023-12
Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models	arXiv	2023-10
FedPrompt: Communication-Efficient and Privacy-Preserving Prompt Tuning in Federated Learning	ICASSP	2023-05

Visual Prompt Tuning

Title	Venue	Year
Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning	CVPR	2024-06
Visual Prompt Based Personalized Federated Learning	TMLR	2024-02
Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation	ICCV	2023-10
Learning Federated Visual Prompt in Null Space for MRI Reconstruction	CVPR	2023-06

Textual-Visual Prompt Tuning

Title	Venue	Year
DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning	CVPR	2024-06
Global and Local Prompts Cooperation via Optimal Transport for Federated Learning	CVPR	2024-06
Federated Text-driven Prompt Generation for Vision-Language Models	ICLR	2024-05
Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning	AAAI	2024-03
Dual Prompt Tuning for Domain-Aware Federated Learning	arXiv	2023-10
PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models - Federated Learning in Age of Foundation Model	TMC	2023-08

Reparameterization-Based

Title	Venue	Year
Improving LoRA in Privacy-preserving Federated Learning	ICLR	2024-05
FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank	LLMAgents@ICLR	2024-05
FedHLT: Efficient Federated Low-Rank Adaption with Hierarchical Language Tree for Multilingual Modeling	WWW	2024-05
FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning	arXiv	2024-04
Towards Building The Federatedgpt: Federated Instruction Tuning	ICASSP	2024-03
Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks	ICASSP	2024-03
Dual-Personalizing Adapter for Federated Foundation Models	arXiv	2024-03
Federated Fine-tuning of Large Language Models under Heterogeneous Language Tasks and Client Resources	arXiv	2024-02
Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models	arXiv	2024-01
SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models	FL@FM-NeurIPS	2023-12
FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models	arXiv	2023-12
Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning	arXiv	2023-12
FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients	arXiv	2023-11
pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning	arXiv	2023-10
Low-Parameter Federated Learning with Large Language Models	arXiv	2023-07

Heterogeneous Resource

Lora

Title	Venue	Year
Federated Fine-tuning of Large Language Models under Heterogeneous Language Tasks and Client Resources	arXiv	2024-02
Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models	arXiv	2024-01
FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients	arXiv	2023-11
pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning	arXiv	2023-10

Split Learning

Title	Venue	Year
Enhancing Storage and Computational Efficiency in Federated Multimodal Learning for Large-Scale Models	ICML	2024-07
Heterogeneous Federated Learning with Splited Language Model	arXiv	2024-03
Federated Split BERT for Heterogeneous Text Classification	IJCNN	2022-12

Sparsification

Title	Venue	Year	GitHub
SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models	FL@FM-NeurIPS	2023-12

Zeroth-Order Optimization

Title	Venue	Year
Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes	ICML	2024-07
FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models	ICML	2024-07
Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning	NAACL	2024-06
On the Convergence of Zeroth-Order Federated Tuning in Large Language Models	arXiv	2024-02
ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning	arXiv	2023-10
Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models	arXiv	2023-10
FwdLLM: Efficient FedLLM using Forward Gradient	arXiv	2023-08

Trustworthiness

IP Protection

Black-Box Tuning

Title	Venue	Year
FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models	ICML	2024-07
Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning	NAACL	2024-06
ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning	arXiv	2023-10
Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models	arXiv	2023-10

Privacy Preservation

Privacy Attack

Title	Venue	Year	GitHub
Analysis of Privacy Leakage in Federated Large Language Models	AISTATS	2024-05
Recovering Private Text in Federated Learning of Language Models	NeurIPS	2022-11

Privacy-Preserving Techniques

Title	Venue	Year
Improving LoRA in Privacy-preserving Federated Learning	ICLR	2024-05
Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning	arXiv	2023-12
Federated Learning of Gboard Language Models with Differential Privacy	ACL	2023-07

Attack Robustness

Poisoning Attack

Title	Venue	Year	GitHub
Unveiling Backdoor Risks Brought by Foundation Models in Heterogeneous Federated Learning	PAKDD	2024-04
Backdoor Threats from Compromised Foundation Models to Federated Learning	FL@FM-NeurIPS	2023-10

Application

Multilingualism

Title	Venue	Year
Generalizable Multilingual Hate Speech Detection on Low Resource Indian Languages using Fair Selection in Federated Learning	NAACL	2024-06
Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages	ICLR	2024-05
Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation	WWW	2024-05
FedHLT: Efficient Federated Low-Rank Adaption with Hierarchical Language Tree for Multilingual Modeling	WWW	2024-05
Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter	ACL	2023-07
Federated Multilingual Models for Medical Transcript Analysis	CHIL	2023-06
Pretrained Models for Multilingual Federated Learning	NAACL	2022-07
FedKC: Federated Knowledge Composition for Multilingual Natural Language Understanding	WWW	2022-04

Speech

Title	Venue	Year
Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks	ICASSP	2024-03
Joint Federated Learning and Personalization for on-Device ASR	ASRU	2023-12
Importance of Smoothness Induced by Optimizers in Fl4Asr: Towards Understanding Federated Learning for End-To-End ASR	ASRU	2023-12
Federated Learning for Speech Recognition: Revisiting Current Trends Towards Large-Scale ASR	FL@FM-NeurIPS	2023-12

Recommendation Systems

Title	Venue	Year
Federated Adaptation for Foundation Model-based Recommendations	IJCAI	2024-08
Navigating the Future of Federated Recommendation Systems with Foundation Models	arXiv	2024-06
Prompt-enhanced Federated Content Representation Learning for Cross-domain Recommendation	WWW	2024-05
Federated Recommendation via Hybrid Retrieval Augmented Generation	arXiv	2024-03
LLM-based Federated Recommendation	arXiv	2024-02
TransFR: Transferable Federated Recommendation with Pre-trained Language Models	arXiv	2024-02

Domain Specific

Title	Venue	Year
Federated Prompt Learning for Weather Foundation Models on Devices	IJCAI	2024-08
Foundation models matter: federated learning for multi-center tuberculosis diagnosis via adaptive regularization and model-contrastive learning	WWW	2024-05
FedFMS: Exploring Federated Foundation Models for Medical Image Segmentation	MICCAI	2024
Prompt Federated Learning for Weather Forecasting: Toward Foundation Models on Meteorological Data	IJCAI	2023-8
FedTherapist: Mental Health Monitoring with User-Generated Linguistic Expressions on Smartphones via Federated Learning	EMNLP	2023-12
FedJudge: Federated Legal Large Language Model	arXiv	2023-09
FEDLEGAL: The First Real-World Federated Learning Benchmark for Legal NLP	ACL	2023-07
Learning Federated Visual Prompt in Null Space for MRI Reconstruction	CVPR	2023-06
Federated Multilingual Models for Medical Transcript Analysis	CHIL	2023-06

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
bibs		bibs
docs		docs
scripts		scripts
.gitignore		.gitignore
LICENSE		LICENSE
README.rst		README.rst

License

lishenghui/awesome-fm-fl

Folders and files

Latest commit

History

Repository files navigation

Awesome-Foundation-Models-with-Federated-Learning

About

Topics

Resources

License

Stars

Watchers

Forks

Languages