Fed Adaptive + Label Variance

What's Changed

• IID vs Dirichlet Selection
• Label Variance weighted averaging
• Fed Adaptive by @hjc2 in #12

FedLoss FedAcc FedFuzz - Lab Runs

FedLoss - weighted averaging based on loss
FedAcc - weighted averaging based on local accuracy
FedFuzz - Fuzzy aggregation

Scripts to generate graphs

Lab Machine capable requirements list

Improved server.app for scaling more strats

File / Data archive of test runs

Updated README

Uneven Partition

Now the partitioning uses a Dirichlet Partitioner

Thus, the samples are no longer evenly distributed among clients

Moving towards adaptive aggregation

Information gathered

• samples per client

• time since last update

• frequency of update

• fuzzy logic to actually make these work

• label variance?

• freshness?

Scripts

Scripts to grab output and make it a nice CSV for chart generation.

grabs all extra aggregated metrics and formats to CSV with round #

frequency

using the data tracking what clients were in what rounds

implement aggregation based on the frequency the clients are included.

last_appearance

server strategy now tracks which clients were included in what rounds

this is used to track how long it has been since the client has been included in a round

this is then used for adaptive aggregation

to do:
fuzzy logic
metric based on how many times included

Stable Custom

Custom still performs FedAvg

Added documentation and removed the main bugs that would cause it possibly to crash immediately.

0.3.0

place for adaptive aggregation is set up.

now I inherit from a base class for logging and vehicle selection purposes for both.

logging_afl.py is the base class for doing simulation of clients entering / exiting and calculating accuracy.

federal_avg.py is kinda the same, but i felt it was clearer to make a new one rather than use logging in case i change something.

federal_agg.py is where the adaptive aggregation will go

wrote some documentation, need to write a bit more.

Poisson Distribution + Accuracy Output

now i need to simulate my adaptive aggregation

INFO :      [SUMMARY]
INFO :      Run finished 2 round(s) in 41.46s
INFO :          History (loss, distributed):
INFO :                  round 1: 73.67386685477362
INFO :                  round 2: 73.6153356552124
INFO :          History (metrics, distributed, evaluate):
INFO :          {'accuracy': [(1, 0.099), (2, 0.1142)], 'count': [(1, 9), (2, 5)]}

FedAvg works

FedAvg Works

FedCustom set up - currently mirrors FedAvg

ran both on

4 server rounds
1 local epoch
2 nodes

running fedAvg Evaluation results - Loss: 283.7736226320267, Accuracy: 0.3456

running fedCustom Evaluation results - Loss: 280.06987047195435, Accuracy: 0.344