Federated Optimization in Heterogeneous Networks
This repository contains the code and experiments for the manuscript:
Federated Learning is a distributed learning paradigm with two key challenges that differentiate it from traditional distributed optimization: (1) significant variability in terms of the systems characteristics on each device in the network (systems heterogeneity), and (2) non-identically distributed data across the network (statistical heterogeneity). In this work, we introduce a framework, FedProx, to tackle heterogeneity in federated networks, both theoretically and empirically.
This repository contains a set of detailed empirical evaluation across a suite of federated datasets. We show that FedProx allows for more robust convergence than FedAvg. In particular, in highly heterogeneous settings, FedProx demonstrates significantly more stable and accurate convergence behavior relative to FedAvg—improving absolute test accuracy by 18.8% on average.
We already includes four synthetic datasets under corresponding folders that are used in the paper. For all datasets, see the
README files in separate
data/$dataset folders for instructions on preprocessing and/or sampling data.
The statistics of real federated datasets are summarized as follows.
pip3 install -r requirements.txt
Run on synthetic federated data
(1) You don't need a GPU to run the synthetic data experiments:
(2) Run the instructions as follows, and the log files will be automatically stored for drawing figures later.
bash run_fedavg.sh synthetic_iid 0 | tee log_synthetic/synthetic_iid_client10_epoch20_mu0 bash run_fedprox.sh synthetic_iid 0 1 | tee log_synthetic/synthetic_iid_client10_epoch20_mu1 bash run_fedavg.sh synthetic_0_0 0 | tee log_synthetic/synthetic_0_0_client10_epoch20_mu0 bash run_fedprox.sh synthetic_0_0 0 1 | tee log_synthetic/synthetic_0_0_client10_epoch20_mu1 bash run_fedavg.sh synthetic_0.5_0.5 0 | tee log_synthetic/synthetic_0.5_0.5_client10_epoch20_mu0 bash run_fedprox.sh synthetic_0.5_0.5 0 1 | tee log_synthetic/synthetic_0.5_0.5_client10_epoch20_mu1 bash run_fedavg.sh synthetic_1_1 0 | tee log_synthetic/synthetic_1_1_client10_epoch20_mu0 bash run_fedprox.sh synthetic_1_1 0 1 | tee log_synthetic/synthetic_1_1_client10_epoch20_mu1
(3) Draw figures to reproduce results on synthetic data
python plot_fig2.py loss # training loss python plot_fig2.py accuracy # testing accuracy python plot_fig2.py dissim # dissimilarity metric
The training loss, testing accuracy and dissimilarity measurement figures are saved as
dissim.pdf respectively, under the current folder where you call
plot_fig2.py. You can check that these figures reproduce the results in Figure 2 in the paper. Make sure to use the default hyper-parameters in
run_fedavg.sh/run_fedprox.sh for synthetic data.
For example, the training loss for synthetic datasets would look like this:
Run on real federated datasets
(1) Specify a GPU id if you want to use GPUs:
Otherwise just run to CPUs [might be slow if testing on non-convex Neural Network models]:
(2) Run on one dataset. First, modify the
run_fedprox.sh scripts, specify the corresponding model to that dataset (choose from
flearn/models/$DATASET/$MODEL.py and use
$MODEL as the model name), specify a log file name, and configure all other parameters such as learning rate (we report all the hyper-parameters in the appendix of the paper):
For example, for all the synthetic data:
python3 -u main.py --dataset=$1 --optimizer='fedavg' \ --learning_rate=0.01 --num_rounds=200 --clients_per_round=10 \ --eval_every=1 --batch_size=10 \ --num_epochs=20 \ --drop_percent=$2 \ --model='mclr'
python3 -u main.py --dataset=$1 --optimizer='fedprox' \ --learning_rate=0.01 --num_rounds=200 --clients_per_round=10 \ --eval_every=1 --batch_size=10 \ --num_epochs=20 \ --drop_percent=$2 \ --model='mclr' \ --mu=$3
mkdir synthetic_1_1 bash run_fedavg.sh synthetic_1_1 0 | tee synthetic_1_1/fedavg_drop0 bash run_fedprox.sh synthetic_1_1 0 0 | tee synthetic_1_1/fedprox_drop0_mu0 bash run_fedprox.sh synthetic_1_1 0 1 | tee synthetic_1_1/fedprox_drop0_mu1 bash run_fedavg.sh synthetic_1_1 0.5 | tee synthetic_1_1/fedavg_drop0.5 bash run_fedprox.sh synthetic_1_1 0.5 0 | tee synthetic_1_1/fedprox_drop0.5_mu0 bash run_fedprox.sh synthetic_1_1 0.5 1 | tee synthetic_1_1/fedprox_drop0.5_mu1 bash run_fedavg.sh synthetic_1_1 0.9 | tee synthetic_1_1/fedavg_drop0.9 bash run_fedprox.sh synthetic_1_1 0.9 0 | tee synthetic_1_1/fedprox_drop0.9_mu0 bash run_fedprox.sh synthetic_1_1 0.9 1 | tee synthetic_1_1/fedprox_drop0.9_mu1
And the accuracy, loss and dissimilarity information will be saved in the log files.
(3) After you collect logs for all the 5 datasets in Figure 1 (synthetic, mnist, femnist, shakespeare, sent140) (the log directories should be
[synthetic_1_1, mnist, femnist, shakespeare, sent140]), run:
python plot_final_e20.py loss
to reproduce results in Figure 1 (the generated figure is called
Note: If you only want to quickly verify the results on the first synthetic dataset, you can modify the
plot_final_e20.py script by changing
range(5) in Line 54 to
range(1), and run
python plot_final_e20.py loss.
Note: It might take a much longer time to run on real datasets than synthetic data because real federated datasets are larger and some of the models are deep neural networks.
See our FedProx manuscript for more details as well as all references.