BENCHMARK_simulation.md

Benchmarking results when using single process-based simulation

Dataset	Model	Accuracy (Exp/Ref)
MNIST	LR	81.9 /
Federated EMNIST	CNN	80.2 / 84.9
fed_CIFAR100	ResNet	34.0 / 44.7
shakespeare (LEAF)	RNN	53.1 /
fed_shakespeare (Google)	RNN	57.1 / 56.9
stackoverflow_nwp	RNN	18.3 / 19.5

Note: Experimental results are the test accuracy of the last communication rounds, while the reference results are the validation results from referenced paper.

Hyper-parameters to reproduce the benchmarking results (non-IID)

MNIST + Logistic Regression

data_args:
  dataset: "mnist"
  partition_method: "hetero"
  partition_alpha: 0.5

model_args:
  model: "lr"
  
train_args:
  federated_optimizer: "FedAvg"
  client_id_list: "[]"
  client_num_in_total: 1000
  client_num_per_round: 10
  comm_round: 200
  epochs: 1
  batch_size: 10
  client_optimizer: sgd
  learning_rate: 0.03

The reference experimental result: https://app.wandb.ai/automl/fedml/runs/ybv29kak

Shakespeare (LEAF) + RNN

data_args:
  dataset: "mnist"
  partition_method: "hetero"
  partition_alpha: 0.5

model_args:
  model: "lr"
  
train_args:
  federated_optimizer: "FedAvg"
  client_id_list: "[]"
  client_num_in_total: 10
  client_num_per_round: 10
  comm_round: 10
  epochs: 1
  batch_size: 10
  client_optimizer: sgd
  learning_rate: 0.8

The experimental result refers to：https://app.wandb.ai/automl/fedml/runs/2al5q5mi

Shakespeare (Google) + RNN

data_args:
  dataset: "shakespeare"
  partition_method: "hetero"
  partition_alpha: 0.5

model_args:
  model: "lr"
  
train_args:
  federated_optimizer: "FedAvg"
  client_id_list: "[]"
  client_num_in_total: 10
  client_num_per_round: 10
  comm_round: 1000
  epochs: 1
  batch_size: 10
  client_optimizer: sgd
  learning_rate: 0.8

The experimental result refers to：https://wandb.ai/automl/fedml/runs/4btyrt0u

Federated EMNIST + CNN

data_args:
  dataset: "femnist"
  partition_method: "hetero"
  partition_alpha: 0.5

model_args:
  model: "lr"
  
train_args:
  federated_optimizer: "FedAvg"
  client_id_list: "[]"
  client_num_in_total: 10
  client_num_per_round: 10
  comm_round: 1000
  epochs: 1
  batch_size: 20
  client_optimizer: sgd
  learning_rate: 0.03

The experimental result refers to：https://wandb.ai/automl/fedml/runs/3lv4gmpz

Fed-CIFAR100 + CNN

data_args:
  dataset: "fed_cifar100"
  partition_method: "hetero"
  partition_alpha: 0.5

model_args:
  model: "resnet18_gn"
  
train_args:
  federated_optimizer: "FedAvg"
  client_id_list: "[]"
  client_num_in_total: 10
  client_num_per_round: 10
  comm_round: 4000
  epochs: 1
  batch_size: 10
  client_optimizer: sgd
  learning_rate: 0.1

The experimental result refers to：https://wandb.ai/automl/fedml/runs/1canbwed

Stackoverflow + Logistic Regression

data_args:
  dataset: "stackoverflow_lr"
  partition_method: "hetero"
  partition_alpha: 0.5

model_args:
  model: "lr"
  
train_args:
  federated_optimizer: "FedAvg"
  client_id_list: "[]"
  client_num_in_total: 10
  client_num_per_round: 10
  comm_round: 2000
  epochs: 1
  batch_size: 10
  client_optimizer: sgd
  learning_rate: 0.03

The experimental result refers to：https://wandb.ai/automl/fedml/runs/3aponqml

Stackoverflow + RNN

data_args:
  dataset: "stackoverflow_nwp"
  partition_method: "hetero"
  partition_alpha: 0.5

model_args:
  model: "rnn"
  
train_args:
  federated_optimizer: "FedAvg"
  client_id_list: "[]"
  client_num_in_total: 10
  client_num_per_round: 10
  comm_round: 2000
  epochs: 1
  batch_size: 10
  client_optimizer: sgd
  learning_rate: 0.03

The experimental result refers to: https://wandb.ai/automl/fedml/runs/7pf2c9r2

CIFAR-10 + ResNet-56

data_args:
  dataset: "cifar10"
  partition_method: "hetero"
  partition_alpha: 0.5

model_args:
  model: "resnet56"
  
train_args:
  federated_optimizer: "FedAvg"
  client_id_list: "[]"
  client_num_in_total: 10
  client_num_per_round: 10
  comm_round: 200
  epochs: 1
  batch_size: 10
  client_optimizer: sgd
  learning_rate: 0.03