Parallel residual Transformer layer #439
Description
Hi,
Is there a way to enable parallel residual similar to HF GPT-neox use_parallel_residual config to speed up training?
@ksivaman If currently not supported do you have any plans to support?
{{ message }}