Skip to content

Commit

Permalink
Merge pull request #9356 from m3ngyang/cluster_trai_pred_trans
Browse files Browse the repository at this point in the history
translate Cluster Training and Prediction
  • Loading branch information
typhoonzero committed Mar 28, 2018
2 parents 25317bd + 68c1994 commit 33614ed
Showing 1 changed file with 12 additions and 1 deletion.
13 changes: 12 additions & 1 deletion doc/v2/faq/cluster/index_en.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,15 @@
Cluster Training and Prediction
###############################

TBD
.. contents::

1. Network connection errors in the log during multi-node cluster training
------------------------------------------------
There are maybe some errors in the log belonging to network connection problem during multi-node cluster training, for example, :code:`Connection reset by peer`.
This kind of error is usually caused by the abnormal exit of a training process in some node, and the other nodes cannot connect with this node any longer. Steps to troubleshoot the problem are as follows:

* Find the first error in the :code:`train.log`, :code:`server.log`, check whether other fault casued the problem, such as FPE, lacking of memory or disk.

* If the first error in server.log says "Address already used", this may be caused by the port conflict of the non-exclusive execution. Connect the sys-admin to check if the current MPI cluster supports jobs submitted with parameter :code:`resource=full`. If the current MPI cluster does not support this parameter, change the server port and try agian.

* If the current MPI cluster does not support exclusive pattern which allows a process to occupy the whole node, ask the administrator to replace or update the this cluster.

0 comments on commit 33614ed

Please sign in to comment.