mpi backend: master should be the last in hostfile

[everpeace]

It is because normal typical usecase uses only workers in distributed learning.

we keep master in hostfile just for the rare scenario which user uses all the pods (master and workers) for distributed learning.

before

Under this hostfile, master pod always attend to mpiexec. But most users want to make only workers attend to it. There is no easy way for users to exclude master in mpiexec.

(master) # cat /kubeflow/chainer-operator/generated/hostfile
example-job-mn-master-hzp66 slots=1
example-job-mn-workerset-ws0-0 slots=1
example-job-mn-workerset-ws1-0 slots=1

after

This is easier for users. user can do in both cases by adjusting mpiexec parameters. With -n 2, only workers will attend to distributed learning. With -n 3, all the pods(master and workers) will attend to it.

(master) # cat /kubeflow/chainer-operator/generated/hostfile
example-job-mn-workerset-ws0-0 slots=1
example-job-mn-workerset-ws1-0 slots=1
example-job-mn-master-hzp66 slots=1

---

This change is 

[everpeace]

@disktnk could you review this?

[disktnk]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: disktnk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [disktnk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment