FL strategies

FedAvg — Communication-Efficient Learning of Deep Networks from Decentralized Data AISTATS 2017. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas.

Improve heterogeneity issues (Non-IID)

FedProx — Federated Optimization in Heterogeneous Networks MLsys 2020 | code

• Statistical Heterogeneity: add regularization in objective to limit the impact of heterogeneous local updates, $\mu=0$ previous, $\mu > 0$ regularized
• System Heterogeneity: incorporate partial updates instead of dropping stragglers

FD and FAug — Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data

• FAug does data augmentation by running a GAN to locally reproduce the data samples of all devices (require uploading local samples)
• FD exchanges not the model parameters but the per-label model output, and see mean model output as teacher's output for distillation

FedBE - FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning ICLR 2021 | code

• Fit a distribution on S locally trained models, and sample M global models in addition to FedAvg and the local models -> in total an ensemble of S + M + 1 models
• Knowledge distillation on the unlabelled server dataset, from the ensemble to the new global model

Fairness

q-FedAvg — Fair Resource Allocation in Federated Learning | code

• q in objective tunes the amount of fairness: $q=0$ -> previous, $q=\infty$ -> maximum fairness

Communication

FedBuff — Federated Learning with Buffered Asynchronous Aggregation

• Buffered Asynchronous Aggregation for scaling and system efficiency, Buffer Size = K

Optimization

FedOpt — Adaptive Federated Optimization

• FedAdam, FedYogi and FedAdagrad

federated SSL

Orchestra

• exploits the federation’s hierarchy to orchestrate a distributed clustering task