standalone Benchmark running the MNIST and Shakespeare

[sjtu-cz]

I can not get the same results when I run the command:
sh run_fedavg_standalone_pytorch.sh 0 10 10 10 shakespeare ./../../../data/shakespeare rnn hetero 100 1 0.8 sgd 0
my results is:
https://wandb.ai/pilgrim_cz/fedml/runs/5jqpw1i4?workspace=user-pilgrim_cz
the benchmart res is https://wandb.ai/automl/fedml/runs/144ey9w6
same problem when I run minst dataset
Is there anything wrong? I did not change any code.

[Elliebababa]

Hi, the link you provide is not accessible. And we have updated the code recently. Did you run the most updated code? Or could you please provide more information so that we can help.

[sjtu-cz]

[sjtu-cz]

ok, I will test the most updated code later, updated 7 hours ago ?

[Elliebababa]

Yes, please have a try.

[chaoyanghe]

@sjtu-cz thanks for your feedback. What's your git version?
Please let us know your results based on the latest code.

[sjtu-cz]

The results are correct now, the problem is caused by parameter passing error in FedML/fedml_experiments/standalone/fedavg/run_fedavg_standalone_pytorch.sh
EPOCH=$10 => EPOCH=${10} ...
This is a small problem，the former epoch is always 0， so the result is not correct.

[chaoyanghe]

@sjtu-cz Great! Happy to know you can reproduce the result.

[am009]

I encountered same problem today. 😭 The error in the script still exists. I really hope it can be corrected so that other people will not meet the same problem again.
I opened a pull request at #180 . Hope it's helpful.

Related link:
https://stackoverflow.com/questions/4912733/how-to-handle-more-than-10-parameters-in-shell