Low accuracy for femnist

[slowbull]

I tried you script with small femnist data:
python3 main.py -dataset femnist -model cnn
the average accuracy is less than 0.1 after 400 rounds.

I also run sgd algorithm:
python3 main.py -dataset femnist -model cnn --minibatch 1.0
the accuracy is still less than 0.1.

Any thoughts?

thanks!

[Enehta]

I have the same problems and I also have results around 50 % for the small sent140 dataset. Could you provide accuracy values incl. used parameters/data set sizes etc for the three datasets, @scaldas? Many thanks.

[gokart23]

@Enehta I was able to reproduce sent140 experiments with the following settings:

• Dataset preprocessing:

leaf/data/sent140/ $> ./preprocess.sh --sf 0.5 -t sample --tf 0.8 -k 3 --spltseed 1549775860

• Model execution:

leaf/models $> python3 main.py -dataset sent140 -model stacked_lstm -lr 0.0003 --clients-per-round 2 --num-rounds 10

For 2 clients, 10 rounds and k=100, the model converges to mean accuracy of ~60% (10th percentile: 25%, 90th percentile: 100%). Can you try this and see if it works out?

[gokart23]

@slowbull @Enehta for the FEMNIST experiments, I have a script in an open PR that should be able to run the experiments (https://github.com/TalwalkarLab/leaf/blob/5aab2848844c82cce82769be1fd4bd14c891c920/femnist.sh). The results generated are qualitatively similar to those reported in the paper. Can you try this and let me know if it works for you?

[Enehta]

@gokart23 I tried both scripts and have not finished all configurations yet, but the FEMNIST script converges around 80 % and the sent140 max. at 60 %. Does that fit to your best results?

[Enehta]

@gokart23 You are creating the data iid, but what is with niid? have you reached good accuracy values there?

[scaldas]

@Enehta: @gokart23 just uploaded some scripts that get a good accuracy with non-iid data (see 7de6548).

[moFang222]

@Enehta Have you reproduced the result of FEMNIST in non-i.i.d. in the original paper?
I also encountered problems with low accuracy!