[distributed training] Program is stuck after the last round of training

[weiyikang]

Running the fedavg on the configure: 20 rounds, 10 epochs, 2 clients, cifar-10 dataset, resnet56, but the program always crashed! The errors as the following:

[weiyikang]

I have running fedavg again use the back end command (nohup), while the same problem also occurs:

nohup sh run_fedavg_distributed_pytorch.sh 4 4 1 4 resnet56 homo 10 5 32 0.001 cifar10 "./../../../data/cifar10" > ./fedavg-resnet-homo-cifar10.txt 2>&1 &

Why the errors always occur in the final ROUNT? The GPUs memory is available while the final ROUND aggregation breaks off.

[chaoyanghe]

Hi, yikang, that's not an error. Your training is successfully finished. The log is somewhat misleading. Let me have a test and modify the logging a little bit.

[chaoyanghe]

INFO:root:#######training########### round_id = 3
INFO:root:(client 642. Local Training Epoch: 0 Loss: 2.123745
INFO:root:#######finished###########
INFO:root:sys.exit(0)
INFO:root:add_model. index = 3
INFO:root:b_all_received = False
INFO:root:add_model. index = 1
INFO:root:b_all_received = False
INFO:root:add_model. index = 0
INFO:root:b_all_received = True
INFO:root:len of self.model_dict[idx] = 4
INFO:root:aggregate time cost: 0
INFO:root:################local_test_on_all_clients : 3
INFO:root:{'training_acc': 0.33828489880643486, 'training_loss': 2.1656083437750917}
INFO:root:{'test_acc': 0.3422873422873423, 'test_loss': 2.165701602372462}
INFO:root:__finish server
INFO:root:sys.exit(0)

[chaoyanghe]

I checked with 4 rounds, it works.

[chaoyanghe]

@weiyikang Hi, I've finished the issue you reported. It is because that we didn't call MPI_Abort() after finishing the training. Please update our code and try again. You can use round 2 and local epoch 1 to have a test. Thank you for your valuable feedback.