fix(nccl): increased nccl timeout

[knzo25]

Summary

During the BEVFusion deployment I made several changes that are not yet in the main branch.
To keep track of the changes, I will split it in several self contained PRs.

In this one, I increased the timeout from nccl, which is triggered by the metrics evaluation taking over 10 minutes (600s seems to be the default value).

Change point

Increased the timeout of nccl

Related error:
https://stackoverflow.com/questions/69693950/error-some-nccl-operations-have-failed-or-timed-out

Note

It applies to all 3d detection config files, but does not affect (negatively) their runtime .

Test performed

• Training with two gpus worked when previously it used to failed during evaluation in the first epoch (note: during the first few epochs evaluation tends to take longer due to a high number of false positives)

[KSeangTan]

I experienced similar NCCL timeout error during my multi-head experiments, thus it's a neccessary and important change