Extend sync_all_reduce API

[KickItLikeShika]

Fixes #1813

Extending sync_all_reduce API, added op arg to allow more reduction operations (SUM, PRODUCT, MAX, MIN)

Check list:

• New tests are added (if a new feature is added)
• New doc strings: description and/or example code are in RST format
• Documentation is updated (if required)

[KickItLikeShika]

Already tested these updates on #1813 and it works fine

[sdesrozis]

Thank you @KickItLikeShika, it looks good. However, unitary tests of this improvement are needed

• test all the combination of op
• catch the error if a wrong op (for instance "var:fakeop", "var:", etc.)

[KickItLikeShika]

I will work on tests once we agree on the implementation

• test all the combination of op

That's already done in torch.distirubted but i can do that in sync_all_reduce(), what do you think?

• catch the error if a wrong op (for instance "var:fakeop", "var:", etc.)

[sdesrozis]

Thank you !

That's already done in torch.distirubted but i can do that in sync_all_reduce(), what do you think?

Tests to update are here

ignite/tests/ignite/metrics/test_metric.py

Line 541 in eca41b9

def _test_distrib_sync_all_reduce_decorator(device):

We must not forgot the update of the documentation too.

[vfdev-5]

Thanks for the PR @KickItLikeShika ! A question about implementation.

[vfdev-5]

LGTM!
@sdesrozis your thoughts and your turn to review :)

[sdesrozis]

@KickItLikeShika Thank you ! LGTM

[KickItLikeShika]

@vfdev-5 @sdesrozis please check the CI failures, we get an assertion error here assert (self.prod.cpu() == (self.prod_nocomp * 5) * (self.prod_nocomp * 5)).all(), But it works fine locally!

[vfdev-5]

@KickItLikeShika do you mean XLA tests? Have you tested it with xla cpu locally ?

[KickItLikeShika]

@vfdev-5 the CPU tests also fail https://github.com/pytorch/ignite/pull/1823/checks?check_run_id=2160328500#step:10:963

[KickItLikeShika]

@vfdev-5 I realized the error, in the prod test i just considered world_size = 2, i will fix that, Thanks!

[KickItLikeShika]

@vfdev-5 can you check this please? https://github.com/pytorch/ignite/pull/1823/checks?check_run_id=2160568207#step:8:1226

[KickItLikeShika]

@sdesrozis can you please check this? https://github.com/pytorch/ignite/pull/1823/checks?check_run_id=2160568207#step:8:1226