Add Flower Baseline: FedAvg+FEMNIST

[charlesbvll]

Paper

Caldas et al., 2018, LEAF: A Benchmark for Federated Settings, FedAvg+FEMNIST

Link

https://arxiv.org/abs/1812.01097

Maybe give motivations about why the paper should be implemented as a baseline.

This benchmark was developed to test federated learning strategies, it is therefore essential to have it as a baseline.

Is there something else you want to add?

No response

Implementation

To implement this baseline, it is recommended to do the following items in that order:

For first time contributors

• Read the first contribution doc
• Complete the Flower tutorial
• Read the Flower Baselines docs to get an overview:
  • https://flower.dev/docs/using-baselines.html
  • https://flower.dev/docs/contributing-baselines.html

Prepare - understand the scope

• Read the paper linked above
• Create the directory structure in Flower Baselines (just the __init__.py files and a README.md)
• Before starting to write code, write down all of the specs of this experiment in a README (dataset, partitioning, model, number of clients, all hyperparameters, …)
• Open a draft PR

Implement - make it work

• Implement some form of dataset loading and partitioning in a separate dataset.py (doesn’t have to match the paper exactly)
• Implement the model in PyTorch
• Write a test that shows that the model has the number of parameters mentioned in the paper
• Implement the federated learning setup outlined in the paper, maybe starting with fewer clients
• Plot accuracy and loss
• Run it and check if the model starts to converge

Align - make it converge

• Implement the exact data partitioning outlined in the paper
• Use the exact hyperparameters outlined in the paper
• Make it converge to roughly the same accuracy that the paper states
• Commit the final hyperparameters and plots
• Mark the PR as ready