Add example for Gloo backend

[johnugeorge]

Related to #7

[royxue]

I just tested this gloo backend could work, no need to change the existed mnist test alot.
Just replace backend, add GPU support, and add destroy_process_group.

If you need help, I could make a PR

[royxue]

However, hmmm which is kinda strange, the gloo can only work with v1alpha2 version, but failed on v1alpha1

[gaocegege]

We do not maintain v1alpha1 anymore, thus it works for us. IMO.

[royxue]

There is still one issue left for gloo backend I was looking into.
The worker pod could reach completed status after destroy_process_group. But the master pod cannot, and will always restart.

This might be related to the pytorch version and NCCL version.

[gaocegege]

We can dive into the issue. If we cannot fix it, we can refer to tf-operator and add a cleanpodpolicy to delete the master pod after the job is finished.

[johnugeorge]

@royxue Why is the master pod restart? Currently RestartPolicy is OnFailure. Does it fail?

[royxue]

yep, it will throw out an error, which would be regarded as failed. I could provide the error info later.

Also, for example, if you didnt call destroy_process_group, gloo would throw the EnForceNotMet error. as shown in (pytorch/pytorch#2530) which would cause all pod restarted.

[royxue]

@johnugeorge
The error is

terminate called after throwing an instance of 'std::system_error'
  what():  Software caused connection abort

[johnugeorge]

Looks like pytorch bug

[royxue]

I think pytorch didnt stop the gloo process correctly when distributed training finished.

[johnugeorge]

This issue is not reproduced in pytorch:04