Accumulate Extensions #761
Conversation
linshokaku
requested review from
asi1024,
emcastillo and
kmaehashi
as code owners
|
/test |
|
/test |
|
|
||
| if self._trigger(manager=manager): | ||
| if self._distributed: | ||
| summary = self._all_reduce_summaries() |
There was a problem hiding this comment.
I wonder if we would like an additional trigger for calling the all_reduce.
Reducing the number of times it is called may improve overall performance but I am not sure if it would be significant.
There was a problem hiding this comment.
The distributed flag is intended to allow values stored for newly reported keys by this accumulate to be the ones that are all_reduced across processes. If want to control the timing of all_reduce with an independent trigger, it can be difficult for users to understand the meaning of the stored values.
In the current implementation, the number of times all_reduce is performed is minimized within the range satisfying the specifications. Therefore, if want to control the number of all_reduce executions, it can be achieved by adjusting the trigger argument.
pytorch_pfn_extras/training/extensions/accumulate/_summary/_average_summary.py
Outdated
Show resolved
Hide resolved
|
/test |
Add Accumulate Extension for more flexible aggregation of what is captured in the report.
Extensions for the following aggregation methods will be added
close #758