Fix Device Event Creation

[eee4017]

PR types

Bug fixes

PR changes

Others

Description

In the Paddle framework, DeviceEvent are currently created using the cudaEventDefault flag. However, these events are exclusively used for synchronization purposes and not for timing measurements. Therefore, it would be more appropriate to create these events with the cudaEventDisableTiming flag. This PR is intended to correct the flag used during CUDA event creation. Additionally, the existing default flag for DeviceEvent is ambiguously set to 0. To improve clarity and prevent misunderstandings, this default flag has been removed. Instead, the flag is now explicitly generated using the GenerateDeviceEventFlag() function.

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[jeng1220]

@Xreki,

This PR should resolve the problem of NCCL not being able to overlap with compute kernels if tensor parallelism is enabled.

[eee4017]

It was mentioned that once the CUDA_DEVICE_MAX_CONNECTIONS flag is set, overlapping operations become entirely unfeasible. This limitation arises from the blocking introduced by the use of cudaEventDefault, since there's an inherent conflict between CUDA_DEVICE_MAX_CONNECTIONS and cudaEventDefault. We can configure the event to utilize cudaEventDisableTiming, so we could use CUDA_DEVICE_MAX_CONNECTIONS.

[GhostScreaming]

Origin Timeline（use paddle.ones to make all_reduce(dx) occupying SM before matmul(dw)）

Timeline with fix PR（remove paddle.ones, and export CUDA_DEVICE_MAX_CONNECTIONS=1）

It fixes the problem of mp_async_allreduce overlap failure.

[GhostScreaming]

This PR fix the problem of mp_async_allreduce overlap failure. PR-CI-ROCM-Compile compilation bug needs to be fixed.

[CLAassistant]

All committers have signed the CLA.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ eee4017
❌ gongweibao

---

gongweibao seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[jeng1220]

@eee4017 ,

PR-CI-ROCM-Compile:

2023-09-23 00:18:56 FAILED: paddle/fluid/platform/CMakeFiles/device_event_test.dir/device_event_test.cc.o
...
2023-09-23 00:18:56 ../paddle/fluid/platform/device_event_test.cc: In member function 'virtual void DeviceEvent_CUDA_Test::TestBody()':
2023-09-23 00:18:56 ../paddle/fluid/platform/device_event_test.cc:89:26: error: no matching function for call to 'paddle::platform::DeviceEvent::DeviceEvent(phi::GPUPlace&)'
2023-09-23 00:18:56    DeviceEvent event(place);

[eee4017]

This branch was cherry-picked by #57656. Close this PR.

[Xreki]

Hi @eee4017 , #57656 是将该PR的修改cherry-pick到incubate/new_frl分支，develop分支仍需求合入该PR。

[GhostScreaming]

line 89, DeviceEvent event(place) -> DeviceEvent event(place, paddle::platform::GenerateDeviceEventFlag().

[Xreki]

LGTM. Great work!